Planet Debian

Subscribe to Planet Debian feed
Planet Debian - http://planet.debian.org/
Updated: 2 hours 7 min ago

Antoine Beaupré: Reliably generating good passwords

9 February, 2017 - 00:00

Passwords are used everywhere in our modern life. Between your email account and your bank card, a lot of critical security infrastructure relies on "something you know", a password. Yet there is little standard documentation on how to generate good passwords. There are some interesting possibilities for doing so; this article will look at what makes a good password and some tools that can be used to generate them.

There is growing concern that our dependence on passwords poses a fundamental security flaw. For example, passwords rely on humans, who can be coerced to reveal secret information. Furthermore, passwords are "replayable": if your password is revealed or stolen, anyone can impersonate you to get access to your most critical assets. Therefore, major organizations are trying to move away from single password authentication. Google, for example, is enforcing two factor authentication for its employees and is considering abandoning passwords on phones as well, although we have yet to see that controversial change implemented.

Yet passwords are still here and are likely to stick around for a long time until we figure out a better alternative. Note that in this article I use the word "password" instead of "PIN" or "passphrase", which all roughly mean the same thing: a small piece of text that users provide to prove their identity.

What makes a good password?

A "good password" may mean different things to different people. I will assert that a good password has the following properties:

  • high entropy: hard to guess for machines
  • transferable: easy to communicate for humans or transfer across various protocols for computers
  • memorable: easy to remember for humans

High entropy means that the password should be unpredictable to an attacker, for all practical purposes. It is tempting (and not uncommon) to choose a password based on something else that you know, but unfortunately those choices are likely to be guessable, no matter how "secret" you believe it is. Yes, with enough effort, an attacker can figure out your birthday, the name of your first lover, your mother's maiden name, where you were last summer, or other secrets people think they have.

The only solution here is to use a password randomly generated with enough randomness or "entropy" that brute-forcing the password will be practically infeasible. Considering that a modern off-the-shelf graphics card can guess millions of passwords per second using freely available software like hashcat, the typical requirement of "8 characters" is not considered enough anymore. With proper hardware, a powerful rig can crack such passwords offline within about a day. Even though a recent US National Institute of Standards and Technology (NIST) draft still recommends a minimum of eight characters, we now more often hear recommendations of twelve characters or fourteen characters.

A password should also be easily "transferable". Some characters, like & or !, have special meaning on the web or the shell and can wreak havoc when transferred. Certain software also has policies of refusing (or requiring!) some special characters exactly for that reason. Weird characters also make it harder for humans to communicate passwords across voice channels or different cultural backgrounds. In a more extreme example, the popular Signal software even resorted to using only digits to transfer key fingerprints. They outlined that numbers are "easy to localize" (as opposed to words, which are language-specific) and "visually distinct".

But the critical piece is the "memorable" part: it is trivial to generate a random string of characters, but those passwords are hard for humans to remember. As xkcd noted, "through 20 years of effort, we've successfully trained everyone to use passwords that are hard for human to remember but easy for computers to guess". It explains how a series of words is a better password than a single word with some characters replaced.

Obviously, you should not need to remember all passwords. Indeed, you may store some in password managers (which we'll look at in another article) or write them down in your wallet. In those cases, what you need is not a password, but something I would rather call a "token", or, as Debian Developer Daniel Kahn Gillmor (dkg) said in a private email, a "high entropy, compact, and transferable string". Certain APIs are specifically crafted to use tokens. OAuth, for example, generates "access tokens" that are random strings that give access to services. But in our discussion, we'll use the term "token" in a broader sense.

Notice how we removed the "memorable" property and added the "compact" one: we want to efficiently convert the most entropy into the shortest password possible, to work around possibly limiting password policies. For example, some bank cards only allow 5-digit security PINs and most web sites have an upper limit in the password length. The "compact" property applies less to "passwords" than tokens, because I assume that you will only use a password in select places: your password manager, SSH and OpenPGP keys, your computer login, and encryption keys. Everything else should be in a password manager. Those tools are generally under your control and should allow large enough passwords that the compact property is not particularly important.

Generating secure passwords

We'll look now at how to generate a strong, transferable, and memorable password. These are most likely the passwords you will deal with most of the time, as security tokens used in other settings should actually never show up on screen: they should be copy-pasted or automatically typed in forms. The password generators described here are all operated from the command line. Password managers often have embedded password generators, but usually don't provide an easy way to generate a password for the vault itself.

The previously mentioned xkcd cartoon is probably a common cultural reference in the security crowd and I often use it to explain how to choose a good passphrase. It turns out that someone actually implemented xkcd author Randall Munroe's suggestion into a program called xkcdpass:

    $ xkcdpass
    estop mixing edelweiss conduct rejoin flexitime

In verbose mode, it will show the actual entropy of the generated passphrase:

    $ xkcdpass -V
    The supplied word list is located at /usr/lib/python3/dist-packages/xkcdpass/static/default.txt.
    Your word list contains 38271 words, or 2^15.22 words.
    A 6 word password from this list will have roughly 91 (15.22 * 6) bits of entropy,
    assuming truly random word selection.
    estop mixing edelweiss conduct rejoin flexitime

Note that the above password has 91 bits of entropy, which is about what a fifteen-character password would have, if chosen at random from uppercase, lowercase, digits, and ten symbols:

    log2((26 + 26 + 10 + 10)^15) = approx. 92.548875

It's also interesting to note that this is closer to the entropy of a fifteen-letter base64 encoded password: since each character is six bits, you end up with 90 bits of entropy. xkcdpass is scriptable and easy to use. You can also customize the word list, separators, and so on with different command-line options. By default, xkcdpass uses the 2 of 12 word list from 12 dicts, which is not specifically geared toward password generation but has been curated for "common words" and words of different sizes.

Another option is the diceware system. Diceware works by having a word list in which you look up words based on dice rolls. For example, rolling the five dice "1 4 2 1 4" would give the word "bilge". By rolling those dice five times, you generate a five word password that is both memorable and random. Since paper and dice do not seem to be popular anymore, someone wrote that as an actual program, aptly called diceware. It works in a similar fashion, except that passwords are not space separated by default:

    $ diceware
    AbateStripDummy16thThanBrock

Diceware can obviously change the output to look similar to xkcdpass, but can also accept actual dice rolls for those who do not trust their computer's entropy source:

    $ diceware -d ' ' -r realdice -w en_orig
    Please roll 5 dice (or a single dice 5 times).
    What number shows dice number 1? 4
    What number shows dice number 2? 2
    What number shows dice number 3? 6
    [...]
    Aspire O's Ester Court Born Pk

The diceware software ships with a few word lists, and the default list has been deliberately created for generating passwords. It is derived from the standard diceware list with additions from the SecureDrop project. Diceware ships with the EFF word list that has words chosen for better recognition, but it is not enabled by default, even though diceware recommends using it when generating passwords with dice. That is because the EFF list was added later on. The project is currently considering making the EFF list be the default.

One disadvantage of diceware is that it doesn't actually show how much entropy the generated password has — those interested need to compute it for themselves. The actual number depends on the word list: the default word list has 13 bits of entropy per word (since it is exactly 8192 words long), which means the default 6 word passwords have 78 bits of entropy:

    log2(8192) * 6 = 78

Both of these programs are rather new, having, for example, entered Debian only after the last stable release, so they may not be directly available for your distribution. The manual diceware method, of course, only needs a set of dice and a word list, so that is much more portable, and both the diceware and xkcdpass programs can be installed through pip. However, if this is all too complicated, you can take a look at Openwall's passwdqc, which is older and more widely available. It generates more memorable passphrases while at the same time allowing for better control over the level of entropy:

    $ pwqgen
    vest5Lyric8wake
    $ pwqgen random=78
    Theme9accord=milan8ninety9few

For some reason, passwdqc restricts the entropy of passwords between the bounds of 24 and 85 bits. That tool is also much less customizable than the other two: what you see here is pretty much what you get. The 4096-word list is also hardcoded in the C source code; it comes from a Usenet sci.crypt posting from 1997.

A key feature of xkcdpass and diceware is that you can craft your own word list, which can make dictionary-based attacks harder. Indeed, with such word-based password generators, the only viable way to crack those passwords is to use dictionary attacks, because the password is so long that character-based exhaustive searches are not workable, since they would take centuries to complete. Changing from the default dictionary therefore brings some advantage against attackers. This may be yet another "security through obscurity" procedure, however: a naive approach may be to use a dictionary localized to your native language (for example, in my case, French), but that would deter only an attacker that doesn't do basic research about you, so that advantage is quickly lost to determined attackers.

One should also note that the entropy of the password doesn't depend on which word list is chosen, only its length. Furthermore, a larger dictionary only expands the search space logarithmically; in other words, doubling the word-list length only adds a single bit of entropy. It is actually much better to add a word to your password than words to the word list that generates it.

Generating security tokens

As mentioned before, most password managers feature a way to generate strong security tokens, with different policies (symbols or not, length, etc). In general, you should use your password manager's password-generation functionality to generate tokens for sites you visit. But how are those functionalities implemented and what can you do if your password manager (for example, Firefox's master password feature) does not actually generate passwords for you?

pass, the standard UNIX password manager, delegates this task to the widely known pwgen program. It turns out that pwgen has a pretty bad track record for security issues, especially in the default "phoneme" mode, which generates non-uniformly distributed passwords. While pass uses the more "secure" -s mode, I figured it was worth removing that option to discourage the use of pwgen in the default mode. I made a trivial patch to pass so that it generates passwords correctly on its own. The gory details are in this email. It turns out that there are lots of ways to skin this particular cat. I was suggesting the following pipeline to generate the password:

    head -c $entropy /dev/random | base64 | tr -d '\n='

The above command reads a certain number of bytes from the kernel (head -c $entropy /dev/random) encodes that using the base64 algorithm and strips out the trailing equal sign and newlines (for large passwords). This is what Gillmor described as a "high-entropy compact printable/transferable string". The priority, in this case, is to have a token that is as compact as possible with the given entropy, while at the same time using a character set that should cause as little trouble as possible on sites that restrict the characters you can use. Gillmor is a co-maintainer of the Assword password manager, which chose base64 because it is widely available and understood and only takes up 33% more space than the original 8-bit binary encoding. After a lengthy discussion, the pass maintainer, Jason A. Donenfeld, chose the following pipeline:

    read -r -n $length pass < <(LC_ALL=C tr -dc "$characters" < /dev/urandom)

The above is similar, except it uses tr to directly to read characters from the kernel, and selects a certain set of characters ($characters) that is defined earlier as consisting of [:alnum:] for letters and digits and [:graph:] for symbols, depending on the user's configuration. Then the read command extracts the chosen number of characters from the output and stores the result in the pass variable. A participant on the mailing list, Brian Candler, has argued that this wastes entropy as the use of tr discards bits from /dev/urandom with little gain in entropy when compared to base64. But in the end, the maintainer argued that reading "reading from /dev/urandom has no [effect] on /proc/sys/kernel/random/entropy_avail on Linux" and dismissed the objection.

Another password manager, KeePass uses its own routines to generate tokens, but the procedure is the same: read from the kernel's entropy source (and user-generated sources in case of KeePass) and transform that data into a transferable string.

Conclusion

While there are many aspects to password management, we have focused on different techniques for users and developers to generate secure but also usable passwords. Generating a strong yet memorable password is not a trivial problem as the security vulnerabilities of the pwgen software showed. Furthermore, left to their own devices, users will generate passwords that can be easily guessed by a skilled attacker, especially if they can profile the user. It is therefore essential we provide easy tools for users to generate strong passwords and encourage them to store secure tokens in password managers.

Note: this article first appeared in the Linux Weekly News.

Alberto García: QEMU and the qcow2 metadata checks

8 February, 2017 - 15:52

When choosing a disk image format for your virtual machine one of the factors to take into considerations is its I/O performance. In this post I’ll talk a bit about the internals of qcow2 and about one of the aspects that can affect its performance under QEMU: its consistency checks.

As you probably know, qcow2 is QEMU’s native file format. The first thing that I’d like to highlight is that this format is perfectly fine in most cases and its I/O performance is comparable to that of a raw file. When it isn’t, chances are that this is due to an insufficiently large L2 cache. In one of my previous blog posts I wrote about the qcow2 L2 cache and how to tune it, so if your virtual disk is too slow, you should go there first.

I also recommend Max Reitz and Kevin Wolf’s qcow2: why (not)? talk from KVM Forum 2015, where they talk about a lot of internal details and show some performance tests.

qcow2 clusters: data and metadata

A qcow2 file is organized into units of constant size called clusters. The cluster size defaults to 64KB, but a different value can be set when creating a new image:

qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G

Clusters can contain either data or metadata. A qcow2 file grows dynamically and only allocates space when it is actually needed, so apart from the header there’s no fixed location for any of the data and metadata clusters: they can appear mixed anywhere in the file.

Here’s an example of what it looks like internally:

In this example we can see the most important types of clusters that a qcow2 file can have:

  • Header: this one contains basic information such as the virtual size of the image, the version number, and pointers to where the rest of the metadata is located, among other things.
  • Data clusters: the data that the virtual machine sees.
  • L1 and L2 tables: a two-level structure that maps the virtual disk that the guest can see to the actual location of the data clusters in the qcow2 file.
  • Refcount table and blocks: a two-level structure with a reference count for each data cluster. Internal snapshots use this: a cluster with a reference count >= 2 means that it’s used by other snapshots, and therefore any modifications require a copy-on-write operation.
Metadata overlap checks

In order to detect corruption when writing to qcow2 images QEMU (since v1.7) performs several sanity checks. They verify that QEMU does not try to overwrite sections of the file that are already being used for metadata. If this happens, the image is marked as corrupted and further access is prevented.

Although in most cases these checks are innocuous, under certain scenarios they can have a negative impact on disk write performance. This depends a lot on the case, and I want to insist that in most scenarios it doesn’t have any effect. When it does, the general rule is that you’ll have more chances of noticing it if the storage backend is very fast or if the qcow2 image is very large.

In these cases, and if I/O performance is critical for you, you might want to consider tweaking the images a bit or disabling some of these checks, so let’s take a look at them. There are currently eight different checks. They’re named after the metadata sections that they check, and can be divided into the following categories:

  1. Checks that run in constant time. These are equally fast for all kinds of images and I don’t think they’re worth disabling.
    • main-header
    • active-l1
    • refcount-table
    • snapshot-table
  2. Checks that run in variable time but don’t need to read anything from disk.
    • refcount-block
    • active-l2
    • inactive-l1
  3. Checks that need to read data from disk. There is just one check here and it’s only needed if there are internal snapshots.
    • inactive-l2

By default all tests are enabled except for the last one (inactive-l2), because it needs to read data from disk.

Disabling the overlap checks

Tests can be disabled or enabled from the command line using the following syntax:

-drive file=hd.qcow2,overlap-check.inactive-l2=on
-drive file=hd.qcow2,overlap-check.snapshot-table=off

It’s also possible to select the group of checks that you want to enable using the following syntax:

-drive file=hd.qcow2,overlap-check.template=none
-drive file=hd.qcow2,overlap-check.template=constant
-drive file=hd.qcow2,overlap-check.template=cached
-drive file=hd.qcow2,overlap-check.template=all

Here, none means that no tests are enabled, constant enables all tests from group 1, cached enables all tests from groups 1 and 2, and all enables all of them.

As I explained in the previous section, if you’re worried about I/O performance then the checks that are probably worth evaluating are refcount-block, active-l2 and inactive-l1. I’m not counting inactive-l2 because it’s off by default. Let’s look at the other three:

  • inactive-l1: This is a variable length check because it depends on the number of internal snapshots in the qcow2 image. However its performance impact is likely to be negligible in all cases so I don’t think it’s worth bothering with.
  • active-l2: This check depends on the virtual size of the image, and on the percentage that has already been allocated. This check might have some impact if the image is very large (several hundred GBs or more). In that case one way to deal with it is to create an image with a larger cluster size. This also has the nice side effect of reducing the amount of memory needed for the L2 cache.
  • refcount-block: This check depends on the actual size of the qcow2 file and it’s independent from its virtual size. This check is relatively expensive even for small images, so if you notice performance problems chances are that they are due to this one. The good news is that we have been working on optimizing it, so if it’s slowing down your VMs the problem might go away completely in QEMU 2.9.
Conclusion

The qcow2 consistency checks are useful to detect data corruption, but they can affect write performance.

If you’re unsure and you want to check it quickly, open an image with overlap-check.template=none and see for yourself, but remember again that this will only affect write operations. To obtain more reliable results you should also open the image with cache=none in order to perform direct I/O and bypass the page cache. I’ve seen performance increases of 50% and more, but whether you’ll see them depends a lot on your setup. In many cases you won’t notice any difference.

I hope this post was useful to learn a bit more about the qcow2 format. There are other things that can help QEMU perform better, and I’ll probably come back to them in future posts, so stay tuned!

Acknowledgments

My work in QEMU is sponsored by Outscale and has been made possible by Igalia and the help of the rest of the QEMU development team.

Shirish Agarwal: Sex, death and nature

8 February, 2017 - 14:56

Yin-Yang-Balance

There is/was a somewhat controversial book by Osho which I read long back ‘Sambhog se Samadhi Ki Aur‘ or the English version ‘From Sex to Superconsciousness

While I can’t say I understand or understood it all, read it about a decade back, the main point shared in the book was that if you are able to achieve bliss/orgasm during sex, you might be able to have a glimpse of super-consciousness.

I had to share the above context as I had gone to a meetup couple of weeks back had gone to a meetup where a friend, Dr. Swati Shome is attempting to write an educational book for teenagers to talk about sex. I did help her a bit in the past I tried to share some of the concerns I had as my generation didn’t have any guidance from parents or teachers. Most of us were left to our own devices which is similar to today’s children as well with the exception that they have the web. You look at both the books, both written in Pune (my home-town) and both talking about the same subject but from so different a view-point. If you see the comments on the meetup page, it really pains to see people’s concerns. I don’t know if there is any solution to the widespread ignorance, myth-making etc. and hence felt a bit sad . Sharing a small clip I had seen few months back.

Just to give a bit of context, the law as has been shared as passed in 2015 has happened after the 2012 Delhi Gang Rape. A part of it is also that the Indian society still frowns upon live-in relationships so in part it may also be a push-back from the conservatives. After all the BJP, a right of the center party has been in power for 2.5 years now so it’s possible that they were part of it. As I don’t have enough knowledge of what the actual case was, who were the litigants and the defendants, the lawyers and the judge involved, I cannot further speculate. If somebody has more info. or link please pass it on. It would be interesting to know if it was a single bench ruling or 3-5 judge bench.

The yin-yan symbol I had shared becomes a bit more apt as in quite a few cultures, including Indian and Japanese, the two are seen as parts of the same coin. One life-giving, the other life-taking or not even taking but converting into something else.

Death

That came few days later when I was reading an article about sleep. The purpose of sleep, is to forget . It was slightly strange and yet interesting article. What disturbed me though, was the bit about the mouse being killed and his brain being sliced. I tried to find many a justification for it, but none I could have peace with. And the crux of that is because the being, the creature’s wilful consent hasn’t been taken. In nature’s eyes humans and mice are one and the same. We don’t get any special passes due to the fact that we are human. A natural disaster doesn’t care whether you are small or big, fat or strong, mouse or wo/man, coward or brave. It’s sheer luck and after disaster preparedness that people and animals get saved or not.

I thought quite a bit that instead of animals being used for scientific experiments, why don’t we use actual humans. While I’m sure PETA supporters probably may have spear-headed this idea for a long time, but it doesn’t mean I can’t come to this realization by myself. After all, it’s not about pandering to a group but rather what I think is right.

Passing the baton to humans does have its own knotty problems though. For any such kind of endeavour, people’s participation and wilful consent would be needed.

While humans can and do give wilful consent, it is difficult problem as you don’t know the situation in which that consent has been taken. We all know about Organ trafficking . Many people especially from lower economic background may be enticed and cheated with the whole economics for science. In most Indian middle and higher-middle classes religion plays a part even though with ‘death’ the body is cremated and is supposed to scatter among the Pancha Mahaboota, the five elements.

I, for one have no hang-ups if some scientist were to slice my brain to find something, provided I’m dead or for that matter any part of the body. If more people thought like that, probably we wouldn’t have to specially grow and then kill lab mice and guinea-pigs to test out theories. Possibly medical innovations would probably be a lot faster than now. Ironically, most medical innovations have happened during wars and continues to do so till date.

Comments, ideas, suggestions and criticisms all are welcome.


Filed under: Miscellenous Tagged: #Death, #Innocence, #Medical Innovatiion, #Medicine, #Murder, #PETA, #Sex, #sleep, #war, education, exploitation, nature

Vincent Bernat: Write your own terminal emulator

8 February, 2017 - 06:30

I was an happy user of rxvt-unicode until I got a laptop with an HiDPI display. Switching from a LoDPI to a HiDPI screen and back was a pain: I had to manually adjust the font size on all terminals or restart them.

VTE is a library to build a terminal emulator using the GTK+ toolkit, which handles DPI changes. It is used by many terminal emulators, like GNOME Terminal, evilvte, sakura, termit and ROXTerm. The library is quite straightforward and writing a terminal doesn’t take much time if you don’t need many features.

Let’s see how to write a simple one.

A simple terminal§

Let’s start small with a terminal with the default settings. We’ll write that in C. Another supported option is Vala.

#include <vte/vte.h>

int
main(int argc, char *argv[])
{
    GtkWidget *window, *terminal;

    /* Initialise GTK, the window and the terminal */
    gtk_init(&argc, &argv);
    terminal = vte_terminal_new();
    window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    gtk_window_set_title(GTK_WINDOW(window), "myterm");

    /* Start a new shell */
    gchar **envp = g_get_environ();
    gchar **command = (gchar *[]){g_strdup(g_environ_getenv(envp, "SHELL")), NULL };
    g_strfreev(envp);
    vte_terminal_spawn_sync(VTE_TERMINAL(terminal),
        VTE_PTY_DEFAULT,
        NULL,       /* working directory  */
        command,    /* command */
        NULL,       /* environment */
        0,          /* spawn flags */
        NULL, NULL, /* child setup */
        NULL,       /* child pid */
        NULL, NULL);

    /* Connect some signals */
    g_signal_connect(window, "delete-event", gtk_main_quit, NULL);
    g_signal_connect(terminal, "child-exited", gtk_main_quit, NULL);

    /* Put widgets together and run the main loop */
    gtk_container_add(GTK_CONTAINER(window), terminal);
    gtk_widget_show_all(window);
    gtk_main();
}

You can compile it with the following command:

gcc -O2 -Wall $(pkg-config --cflags --libs vte-2.91) term.c -o term

And run it with ./term:

More features§

From here, you can have a look at the documentation to alter behavior or add more features. Here are three examples.

Colors§

You can define the 16 basic colors with the following code:

#define CLR_R(x)   (((x) & 0xff0000) >> 16)
#define CLR_G(x)   (((x) & 0x00ff00) >>  8)
#define CLR_B(x)   (((x) & 0x0000ff) >>  0)
#define CLR_16(x)  ((double)(x) / 0xff)
#define CLR_GDK(x) (const GdkRGBA){ .red = CLR_16(CLR_R(x)), \
                                    .green = CLR_16(CLR_G(x)), \
                                    .blue = CLR_16(CLR_B(x)), \
                                    .alpha = 0 }
vte_terminal_set_colors(VTE_TERMINAL(terminal),
    &CLR_GDK(0xffffff),
    &(GdkRGBA){ .alpha = 0.85 },
    (const GdkRGBA[]){
        CLR_GDK(0x111111),
        CLR_GDK(0xd36265),
        CLR_GDK(0xaece91),
        CLR_GDK(0xe7e18c),
        CLR_GDK(0x5297cf),
        CLR_GDK(0x963c59),
        CLR_GDK(0x5E7175),
        CLR_GDK(0xbebebe),
        CLR_GDK(0x666666),
        CLR_GDK(0xef8171),
        CLR_GDK(0xcfefb3),
        CLR_GDK(0xfff796),
        CLR_GDK(0x74b8ef),
        CLR_GDK(0xb85e7b),
        CLR_GDK(0xA3BABF),
        CLR_GDK(0xffffff)
}, 16);

While you can’t see it on the screenshot1, this also enables background transparency.

Miscellaneous settings§

VTE comes with many settings to change the behavior of the terminal. Consider the following code:

vte_terminal_set_scrollback_lines(VTE_TERMINAL(terminal), 0);
vte_terminal_set_scroll_on_output(VTE_TERMINAL(terminal), FALSE);
vte_terminal_set_scroll_on_keystroke(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_rewrap_on_resize(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_mouse_autohide(VTE_TERMINAL(terminal), TRUE);

This will:

  • disable the scrollback buffer,
  • not scroll to the bottom on new output,
  • scroll to the bottom on keystroke,
  • rewrap content when terminal size change, and
  • hide the mouse cursor when typing.
Update the window title§

An application can change the window title using XTerm control sequences (for example, with printf "\e]2;${title}\a"). If you want the actual window title to reflect this, you need to define this function:

static gboolean
on_title_changed(GtkWidget *terminal, gpointer user_data)
{
    GtkWindow *window = user_data;
    gtk_window_set_title(window,
        vte_terminal_get_window_title(VTE_TERMINAL(terminal))?:"Terminal");
    return TRUE;
}

Then, connect it to the appropriate signal, in main():

g_signal_connect(terminal, "window-title-changed", 
    G_CALLBACK(on_title_changed), GTK_WINDOW(window));
Final words§

I don’t need much more as I am using tmux inside each terminal. In my own copy, I have also added the ability to complete a word using ones from the current window or other windows (also known as dynamic abbrev expansion). This requires to implement a terminal daemon to handle all terminal windows with one process, similar to urxvtcd.

While writing a terminal “from scratch”2 suits my need, it may not be worth it. evilvte is quite customizable and can be lightweight. Consider it as a first alternative. Honestly, I don’t remember why I didn’t pick it. You should also note that the primary goal of VTE is to be a library to support GNOME Terminal. Notably, if a feature is not needed for GNOME Terminal, it won’t be added to VTE. If it already exists, it will likely to be deprecated and removed.

  1. Transparency is handled by the composite manager (Compton, in my case). 

  2. For some definition of “scratch” since the hard work is handled by VTE

Carl Chenet: The Gitlab database incident and the Backup Checker project

8 February, 2017 - 06:00

The Gitlab.com database incident of 2017/01/31 and the resulting data loss reminded everyone (at least for the next days) how it’s easy to lose data, even when you think all your systems are safe.

Being really interested by the process of backing up data, I read with interest the report (kudos to the Gitlab company for being so transparent about it) and I was soooo excited to find the following sentence:

Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.

Whoa, guys! I’m so sorry for you about the data loss, but from my point of view I was so excited to find a big FOSS company publicly admitting and communicating about a perfect use case for the Backup Checker project, a Free Software I’ve been writing these last years.

Data loss: nobody cares before, everybody cries after

Usually people don’t care about the backups. It’s a serious business for web hosters and the backup team from big companies but otherwise and in other places, nobody cares.

Usually everybody agrees about how backups are important but few people make them or install an automatized system to create backups and the day before, nobody verifies they are usable. The reason is obvious: it’s totally boring, and in some cases e.g for large archives, difficult.

Because verifying backups is boring for humans, I launched the Backup Checker project in order to automatize this task.

Backup Checker offers a wide range of features, checking lots of different archives (tar.{gz,bz2,xz}, zip, tree of files and offer lots of different tests (hash sum, size {equal, smaller/greater than}, unix rights, …,). Have a look at the official documentation for a exhaustive list of features and possible tests.

Automatize the controls of your backups with Backup Checker

Checking your backups means to describe in a configuration file how a backup should be, e.g a gzipped database dump. You usually know about what size the archive is going to be, what the owner and the group owner should be.

Even easier, with Backup Checker you can generate this list of criterias from an actual archive, and remove uneeded criterias to create a template you can re-use for different kind of archives.

Ok, 2 minutes of your time for a real word example, I use an existing database sql dump in an tar.gz archive to automatically create the list describing this backup:

$ backupchecker -G database-dump.tar.gz
$ cat database-dump.list
[archive]
mtime| 1486480274.2923253

[files]
database.sql| =7854803 uid|1000 gid|1000 owner|chaica group|chaica mode|644 type|f mtime|1486480253.0

Now, just remove parameters too precise from this list to get a backup template. Here is a possible result:

[files]
database.sql| >6m uid|1000 gid|1000 mode|644 type|f

We define here a template for the archive, meaning that the database.sql file in the archive should have a size greater than 6 megabytes, be owned by the user with the uid of 1000 and the group with a gid of 1000, this file should have the mode 644 and be a regular file. In order to use a template instead of the complete list, you also need to remove the sha512 from the .conf file.

Pretty easy hmm? Ok, just for fun, lets replicate the part of the Gitlab.com database incident mentioned above and write an archive with an empty sql dump inside an archive:

$ touch /tmp/database.sql && \
tar zcvf /tmp/database-dump.tar.gz /tmp/database.sql && \
cp /tmp/database-dump.tar.gz .

Now we launch Backup Checker with the previously created template. If you didn’t change the name of database-dump.list file, the command should only be:

$ backupchecker -C database-dump.conf
$ cat a.out 
WARNING:root:1 file smaller than expected while checking /tmp/article-backup-checker/database-dump.tar.gz: 
WARNING:root:database.sql size is 0. Should have been bigger than 6291456.

The automatized controls of Backup Checker trigger a warning in the log file. The empty sql dump has been identified inside the archive.

A step further

As you could read in this article, verifying some of your backups is not a time consuming task, given the fact you have a FOSS project dedicated to this task, with an easy way to realize a template of your backups and to use it.

This article provided a really simple example of such a use case, the Backup Checker has lots of features to offer when verifying your backups. Read the official documentation for more complete descriptions of the available possibilities.

Data loss, especially for projets storing user data is always a terrible event in the life of an organization. Lets try to learn from mistakes which could happen to anyone and build better backup systems.

More information about the Backup Checker project

 

 

Craig Small: WordPress 4.7.2

8 February, 2017 - 03:53

When WordPress originally announced their latest security update, there were three security fixes. While all security updates can be serious, they didn’t seem too bad. Shortly after, they updated their announcement with a fourth and more serious security problem.

I have looked after the Debian WordPress package for a while. This is the first time I have heard people actually having their sites hacked almost as soon as this vulnerability was announced.

If you are running WordPress 4.7 or 4.7.1, your website is vulnerable and there are bots out there looking for it. You should immediately upgrade to 4.7.2 (or, if there is a later 4.7.x version to that).  There is now updated Debian wordpress version 4.7.2 packages for unstable, testing and stable backports.

For stable, you are on a patched version 4.1 which doesn’t have this specific vulnerability (it was introduced in 4.7) but you should be using 4.1+dfsg-1+deb8u12 which has the fixes found in 4.7.1 ported back to 4.1 code.

Bits from Debian: DebConf17: Call for Proposals

8 February, 2017 - 03:00

The DebConf Content team would like to Call for Proposals for the DebConf17 conference, to be held in Montreal, Canada, from August 6 through August 12, 2017.

You can find this Call for Proposals in its latest form at: https://debconf17.debconf.org/cfp

Please refer to this URL for updates on the present information.

Submitting an Event

Submit an event proposal and describe your plan. Please note, events are not limited to traditional presentations or informal sessions (BoFs). We welcome submissions of tutorials, performances, art installations, debates, or any other format of event that you think would be beneficial to the Debian community.

Please include a short title, suitable for a compact schedule, and an engaging description of the event. You should use the field "Notes" to provide us information such as additional speakers, scheduling restrictions, or any special requirements we should consider for your event.

Regular sessions may either be 20 or 45 minutes long (including time for questions), other kinds of sessions (like workshops) could have different durations. Please choose the most suitable duration for your event and explain any special requests.

You will need to create an account on the site, to submit a talk. We'd encourage Debian account holders (e.g. DDs) to use Debian SSO when creating an account. But this isn't required for everybody, you can sign up with an e-mail address and password.

Timeline

The first batch of accepted proposals will be announced in April. If you depend on having your proposal accepted in order to attend the conference, please submit it as soon as possible so that it can be considered during this first evaluation period.

All proposals must be submitted before Sunday 4 June 2017 to be evaluated for the official schedule.

Topics and Tracks

Though we invite proposals on any Debian or FLOSS related subject, we have some broad topics on which we encourage people to submit proposals, including:

  • Blends
  • Debian in Science
  • Cloud and containers
  • Social context
  • Packaging, policy and infrastructure
  • Embedded
  • Systems administration, automation and orchestration
  • Security

You are welcome to either suggest more tracks, or become a coordinator for any of them; please refer to the Content Tracks wiki page for more information on that.

Code of Conduct

Our event is covered by a Code of Conduct designed to ensure everyone's safety and comfort. The code applies to all attendees, including speakers and the content of their presentations. For more information, please see the Code on the Web, and do not hesitate to contact us at content@debconf.org if you have any questions or are unsure about certain content you'd like to present.

Video Coverage

Providing video of sessions amplifies DebConf achievements and is one of the conference goals. Unless speakers opt-out, official events will be streamed live over the Internet to promote remote participation. Recordings will be published later under the DebConf license, as well as presentation slides and papers whenever available.

DebConf would not be possible without the generous support of all our sponsors, especially our Platinum Sponsor Savoir-Faire Linux. DebConf17 is still accepting sponsors; if you are interested, or think you know of others who would be willing to help, please get in touch!

In case of any questions, or if you wanted to bounce some ideas off us first, please do not hesitate to reach out to us at content@debconf.org.

We hope to see you in Montreal!

The DebConf team

Jonathan McDowell: GnuK on the Maple Mini

8 February, 2017 - 01:34

Last weekend, as a result of my addiction to buying random microcontrollers to play with, I received some Maple Minis. I bought the Baite clone direct from AliExpress - so just under £3 each including delivery. Not bad for something that’s USB capable, is based on an ARM and has plenty of IO pins.

I’m not entirely sure what my plan is for the devices, but as a first step I thought I’d look at getting GnuK up and running on it. Only to discover that chopstx already has support for the Maple Mini and it was just a matter of doing a ./configure --vidpid=234b:0000 --target=MAPLE_MINI --enable-factory-reset ; make. I’d hoped to install via the DFU bootloader already on the Mini but ended up making it unhappy so used SWD by following the same steps with OpenOCD as for the FST-01/BusPirate. (SWCLK is D21 and SWDIO is D22 on the Mini). Reset after flashing and the device is detected just fine:

usb 1-1.1: new full-speed USB device number 73 using xhci_hcd
usb 1-1.1: New USB device found, idVendor=234b, idProduct=0000
usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.1: Product: Gnuk Token
usb 1-1.1: Manufacturer: Free Software Initiative of Japan
usb 1-1.1: SerialNumber: FSIJ-1.2.3-87155426

And GPG is happy:

$ gpg --card-status
Reader ...........: 234B:0000:FSIJ-1.2.3-87155426:0
Application ID ...: D276000124010200FFFE871554260000
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 87155426
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

While GnuK isn’t the fastest OpenPGP smart card implementation this certainly seems to be one of the cheapest ways to get it up and running. (Plus the fact that chopstx already runs on the Mini provides me with a useful basis for other experimentation.)

Olivier Berger: Making Debian stable/jessie images for OpenStack with bootstrap-vz and cloud-init

8 February, 2017 - 00:09

I’m investigating the creation of VM images for different virtualisation solutions.

Among the target platforms is a destop as a service platform based on an OpenStack public cloud.

We’ve been working with bootstrap-vz for creating VMs for Vagrant+VirtualBox so I wanted to test its use for OpenStack.

There are already pre-made images available, including official Debian ones, but I like to be able to re-create things instead of depending on some external magic (which also means to be able to optimize, customize and avoid potential MitM, of course).

It appears that bootstrap-vz can be used with cloud-init provided that some bits of config are specified.

In particular the cloud_init plugin of bootstrap-vz requires a metadata_source set to “NoCloud, ConfigDrive, OpenStack, Ec2“. Note we explicitely spell it ‘OpenStack‘ and not ‘Openstack‘ as was mistakenly done in the default Debian cloud images (see https://bugs.debian.org/854482).

The following snippet of manifest provides the necessary bits :

---
name: debian-{system.release}-{system.architecture}-{%Y}{%m}{%d}
provider:
  name: kvm
  virtio_modules:
  - virtio_pci
  - virtio_blk
bootstrapper:
  workspace: /target
  # create or reuse a tarball of packages
  tarball: true
system:
  release: jessie
  architecture: amd64
  bootloader: grub
  charmap: UTF-8
  locale: en_US
  timezone: UTC
volume:
  backing: raw
  partitions:
    #type: gpt
    type: msdos
    root:
      filesystem: ext4
      size: 4GiB
    swap:
      size: 512MiB
packages:
  # change if another mirror is closer
  mirror: http://ftp.fr.debian.org/debian/
plugins:
  root_password:
    password: whatever
  cloud_init:
    username: debian
    # Note we explicitely spell it 'OpenStack' and not 'Openstack' as done in the default Debian cloud images (see https://bugs.debian.org/854482)
    metadata_sources: NoCloud, ConfigDrive, OpenStack, Ec2
  # admin_user:
  #   username: Administrator
  #   password: Whatever
  minimize_size:
    # reduce the size by around 250 Mb
    zerofree: true

I’ve tested this with the bootstrap-vz version in stretch/testing (0.9.10+20170110git-1) for creating jessie/stable image, which were booted on the OVH OpenStack public cloud. YMMV.

Hope this helps

Sven Hoexter: Dell Latitude E7470 hold and mark with upper left touchpad button

7 February, 2017 - 18:55

Recently some of my coworkers and I experienced an issue with using the upper left touchpad button on our Dell Latitude E7470 and similar laptops (E5xxx from the current generation). Some time in January we could no longer hold down this button and select text with the touchpad. Using the left button below the touchpad still worked. This hit my coworker running Fedora and myself running Debian/stretch. So I first thought that it's likely a libinput issue (same version in Debian/stretch and Fedora and I recently pulled that in as an update), somehow blacklisting the upper left key because it's connected to the trackpoint. So I filled #99594 upstream. While this was not very helpful at first, and according to Peter very unlikely to be related to libinput, another coworker using Debian/jessie found this issue to hit him when he upgraded the backports kernel in use from 4.8 to 4.9. That finally led to the conclusion that it's a bug in the Linux alps driver, which is already fixed in 4.10 and probably 4.9.6.

Until the Debian kernel team pulls in a fresh 4.9 point release I'm using 4.10-rc6 from experimental. For Debian/jessie + backports kernel user it might be more convenient to just stay at 4.8 in case this issue annoys you.

Kudos to Peter, Benjamin, TW and WW for the help in locating the origin of this issue!

Lessons learned:

  • I should've started with the painful downgrade of xorg and libinput via snapshot.d.o before opening the bugreport.
  • A lot more of the touchpad related hardware support is nowadays in the kernel and not in the xorg layer. Either that was just my personal historic misunderstanding, or it was different 10 years ago.
  • There is an interesting set of slides from Benjamin related to debuging input device issues.

Junichi Uekawa: According to annual health check my weight has not increased the last year.

7 February, 2017 - 18:36
According to annual health check my weight has not increased the last year. Hopefully that's because of going to the gym.

Sean Whitton: reclaiming conversation

7 February, 2017 - 09:52

On Friday night I attended a talk by Sherry Turkle called “Reclaiming Conversation: The Power of Talk in a Digital Age”. Here are my notes.

Turkle is an anthropologist who interviews people from different generations about their communication habits. She has observed cross-generational changes thanks to (a) the proliferation of instant messaging apps such as WhatsApp and Facebook Messenger; and (b) fast web searching from smartphones.

Her main concern is that conversation is being trivialised. Consider six or seven college students eating a meal together. Turkle’s research has shown that the etiquette among such a group has shifted such that so long as at least three people are engaged in conversation, others at the table feel comfortable turning their attention to their smartphones. But then the topics of verbal conversation will tend away from serious issues – you wouldn’t talk about your mother’s recent death if anyone at the table was texting.

There are also studies that purport to show that the visibility of someone’s smartphone causes them to take a conversation less seriously. The hypothesis is that the smartphone is a reminder of all the other places they could be, instead of with the person they are with.

A related cause of the trivialisation of conversation is that people are far less willing to make themselves emotionally vulnerable by talking about serious matters. People have a high degree of control over the interactions that take place electronically (they can think about their reply for much longer, for example). Texting is not open-ended in the way a face-to-face conversation is. People are unwilling to give up this control, so they choose texting over talking.

What is the upshot of these two respects in which conversation is being trivialised? Firstly, there are psycho-social effects on individuals, because people are missing out on opportunities to build relationships. But secondly, there are political effects. Disagreeing about politics immediately makes a conversation quite serious, and people just aren’t having those conversations. This contributes to polarisation.

Note that this is quite distinct from the problems of fake news and the bubbling effects of search engine algorithms, including Facebook’s news feed. It would be much easier to tackle fake news if people talked about it with people around them who would be likely to disagree with them.

Turkle understands connection as a capacity for solitude and also for conversation. The drip feed of information from the Internet prevents us from using our capacity for solitude. But then we fail to develop a sense of self. Then when we finally do meet other people in real life, we can’t hear them because we just use them to try to establish a sense of self.

Turkle wants us to be more aware of the effects that our smartphones can have on conversations. People very rarely take their phone out during a conversation because they want to escape from that conversation. Instead, they think that the phone will contribute to that conversation, by sharing some photos, or looking up some information online. But once the phone has come out, the conversation almost always takes a turn for the worse. If we were more aware of this, we would have access to deeper interactions.

A further respect in which the importance of conversation is being downplayed is in the relationships between teachers and students. Students would prefer to get answers by e-mail than build a relationship with their professors, but of course they are expecting far too much of e-mail, which can’t teach them in the way interpersonal contact can.

All the above is, as I said, cross-generational. Something that is unique to millenials and below is that we seek validation for the way that we feel using social media. A millenial is not sure how they feel until they send a text or make a broadcast (this makes them awfully dependent on others). Older generations feel something, and then seek out social interaction (presumably to share, but not in the social media sense of ‘share’).

What does Turkle think we can do about all this? She had one positive suggestion and one negative suggestion. In response to student or colleague e-mails asking for something that ought to be discussed face-to-face, reply “I’m thinking.” And you’ll find they come to you. She doesn’t want anyone to write “empathy apps” in response to her findings. For once, more tech is definitely not the answer.

Turkle also made reference to the study reported here and here and here.

Joachim Breitner: Why prove programs equivalent when your compiler can do that for you?

7 February, 2017 - 07:38

Last week, while working on CodeWorld, via a sequence of yak shavings, I ended up creating a nicely small library that provides Control.Applicative.Succs, a new applicative functor. And because I am trying to keep my Haskell karma good1, I wanted to actually prove that my code fulfills the Applicative and Monad laws.

This led me to inserted writing long comments into my code, filled with lines like this:

The second Applicative law:

  pure (.) <*> Succs u us <*> Succs v vs <*> Succs w ws
= Succs (.) [] <*> Succs u us <*> Succs v vs <*> Succs w ws
= Succs (u .) (map (.) us) <*> Succs v vs <*> Succs w ws
= Succs (u . v) (map ($v) (map (.) us) ++ map (u .) vs) <*> Succs w ws
= Succs (u . v) (map (($v).(.)) us ++ map (u .) vs) <*> Succs w ws
= Succs ((u . v) w) (map ($w) (map (($v).(.)) us ++ map (u .) vs) ++ map (u.v) ws)
= Succs ((u . v) w) (map (($w).($v).(.)) us ++ map (($w).(u.)) vs ++ map (u.v) ws)
= Succs (u (v w)) (map (\u -> u (v w)) us ++ map (\v -> u (v w)) vs ++ map (\w -> u (v w)) ws)
= Succs (u (v w)) (map ($(v w)) us ++ map u (map ($w) vs ++ map v ws))
= Succs u us <*> Succs (v w) (map ($w) vs ++ map v ws)
= Succs u us <*> (Succs v vs <*> Succs w ws)

Honk if you have done something like this before!

I proved all the laws, but I was very unhappy. I have a PhD on something about Haskell and theorem proving. I have worked with Isabelle, Agda and Coq. Both Haskell and theorem proving is decades old. And yet, I sit here, and tediously write manual proofs by hand. Is this really the best we can do?

Of course I could have taken my code, rewritten it in, say, Agda, and proved it correct there. But (right now) I don’t care about Agda code. I care about my Haskell code! I don’t want to write it twice, worry about copying mistakes and mismatchs in semantics, and have external proofs to maintain. Instead, I want to prove where I code, and have the proofs checked together with my code!

Then it dawned to me that this is, to some extent, possible. The Haskell compiler comes with a sophisticated program transformation machinery, which is meant to simplify and optimize code. But it can also be used to prove Haskell expressions to be equivalent! The idea is simple: Take two expressions, run both through the compiler’s simplifier, and check if the results are the same. If they are, then the expressions are, as far as the compiler is concerned, equivalent.

A handful of hours later, I was able to write proof tasks like

app_law_2 = (\ a b (c::Succs a) -> pure (.) <*> a <*> b <*> c)
        === (\ a b c -> a <*> (b <*> c))

and others into my source file, and the compiler would tell me happily:

[1 of 1] Compiling Successors       ( Successors.hs, Successors.o )
GHC.Proof: Proving getCurrent_proof1 …
GHC.Proof: Proving getCurrent_proof2 …
GHC.Proof: Proving getCurrent_proof3 …
GHC.Proof: Proving ap_star …
GHC.Proof: Proving getSuccs_proof1 …
GHC.Proof: Proving getSuccs_proof2 …
GHC.Proof: Proving getSuccs_proof3 …
GHC.Proof: Proving app_law_1 …
GHC.Proof: Proving app_law_2 …
GHC.Proof: Proving app_law_3 …
GHC.Proof: Proving app_law_4 …
GHC.Proof: Proving monad_law_1 …
GHC.Proof: Proving monad_law_2 …
GHC.Proof: Proving monad_law_3 …
GHC.Proof: Proving return_pure …
GHC.Proof proved 15 equalities

This is how I want to prove stuff about my code!

Do you also want to prove stuff about your code? I packaged this up as a GHC plugin in the Haskell library ghc-proofs (not yet on Hackage). The README of the repository has a bit more detail on how to use this plugin, how it works, what its limitations are and where this is heading.

This is still only a small step, but finally there is a step towards low threshold program equivalence proofs in Haskell.

  1. Or rather recover my karma after such abominations such as ghc-dup, seal-module or ghc-heap-view.

Martin Pitt: Migrated blog from WordPress to Hugo

7 February, 2017 - 03:04

My WordPress blog got hacked two days ago and now twice today. This morning I purged MySQL and restored a good backup from three days ago, changed all DB and WordPress passwords (both the old and new ones were long and autogenerated ones), but not even an hour after the redeploy the hack was back. (It can still be seen on Planet Debian and Planet Ubuntu. Neither the Apache logs nor the Journal had anything obvious, nor were there any new files in global or user www directories, so I’m a bit stumped how this happened. Certainly not due to bruteforcing a password, that would both have shown in the logs and also have triggered ban2fail, so this looks like an actual vulnerability.

I upgraded to WordPress 4.7.1 a few days ago, and apparently 4.7.2 fixes a few vulnerabilities, although all of them don’t sound like they would match my situation. jessie-backports is still at 4.7.1, so I missed that update. But either way, all WordPress blogs hosted on my server are down for the time being.

I took this as motivation to finally migrate to something more robust. WordPress has tons of features that I never need, and also a lot of overhead (dynamic generation, MySQL, its own user/passwords, etc.). I had a look around, and it seems Hugo and Blogofile are nice contenders – no privileges, no database, outputting static files, input is Markdown (so much nicer to type than HTML!), and maintaining your blog in git and previewing the changes on my local laptop are straightforward. I happened to try Hugo first, and like it enough to give it an extended try – you have plenty of themes to choose from and they are straightforward to customize, so I don’t need to spend a lot of time learning and crafting CSS.

I ran the WordPress to Hugo Exporter, and it produced remarkable results – fairly usable HTML → Markdown and metadata conversion, it keeps all the original URLs, and it’s painless to use. Nicely done!

So here it is, on to a much more secure server now! \o/

Wouter Verhelst: FOSDEM 2017 is finished...

6 February, 2017 - 20:53

... but that doesn't mean the work is over.

One big job that needs to happen after the conference is to review and release the video recordings that were made. With several hundreds of videos to be checked and only a handful of people with the ability to do so, review was a massive job that for the past three editions took several months; e.g., in 2016 the last video work was done in July, when the preparation of the 2017 edition had already started.

Obviously this is suboptimal, and therefore another solution was required. After working on it for quite a while (in my spare time), I came up with SReview, a video review and transcoding system written in Perl.

An obvious question that could be asked is why I wrote yet another system, though, and did not use something that already existed. The short answer to that is "because what's there did not exactly do what I wanted to". The somewhat longer answer also involves the fact that I felt like writing something from scratch.

The full story, however, is this: there isn't very much out there, and what does exist is flawed in some ways. I am aware of three other review systems that are or were used by other conferences:

  1. A bunch of shell scripts that were written by the DebConf video team and hooked into the penta database. Nobody but DebConf ever used it. It allowed review via an NFS share and a webinterface, and required people to watch .dv files directly from the filesystem in a media player. For this and other reasons, it could only ever be used from the conference itself. If nothing else, that final limitation made it impossible for FOSDEM to use it, but even if that wasn't the case it was still too basic to ever be useful for a conference the size of FOSDEM.
  2. A review system written by the CCC "voc" team. I've never actually seen it in use, but I've heard people describe it. It involves a complicated setup of NFS (or was it HTTP?) servers, short MPEG-4 transport stream segments, a FUSE filesystem, and kdenlive, which took someone several days to set up as an experiment back at DebConf15. Critically, important parts of it are also not licensed as free software, which to me rules it out for a tool in support of FOSDEM. Even if that wasn't the case, however, I'm still not sure it would be ideal; this system requires intimate knowledge of how it works from its user, which makes it harder for us to crowdsource the review to the speaker, as I had planned to.
  3. Veyepar. This one gets many things right, and we used it for video review at DebConf from DebConf14 onwards, as well as FOSDEM 2014 (but not 2015 or 2016). Unfortunately, it also gets many things wrong. Most of these can be traced back to the fact that Carl, as he freely admits, is not a programmer; he's more of a sysadmin type who also manages to cobble together a few scripts now and then. Some of the things it gets wrong are minor issues that would theoretically be fixable with a minimal amount of effort; others would be more involved. It is also severely underdocumented, and so as a result it is rather tedious for someone not very familiar with the system to be able to use it. On a more personal note, veyepar is also written in the wrong language, so while I might have spent some time improving it, I ended up starting from scratch.

Something all these systems have in common is that they try to avoid postprocessing as much as possible. This only makes sense; if you have to deal with loads and loads of video recordings, having to do too much postprocessing only ensures that it won't get done...

Despite the issues that I have with it, I still think that veyepar is a great system, and am not ashamed to say that SReview borrows many ideas and concepts from it. However, it does things differently in some areas, too:

  1. A major focus has been on making the review form be as easy to use as possible. While there is still room for improvement (and help would certainly be welcome in that area from someone with more experience in UI design than me), I think the SReview review form is much easier to use than the veyepar one (which has so many options that it's pretty hard to understand sometimes).
  2. SReview assumes that as soon as there are recordings in a given room sufficient to fill all the time that a particular event in that room was scheduled for, the whole event is available. It will then generate a first rough cut, and send a notification to the speaker in question, as well as the people who organized the devroom. The reviewer will then almost certainly be required to request a second (and possibly third or fourth) cut, but I think the advantage of that is that it makes the review workflow be more intuitive and easier to understand.
  3. Where veyepar requires one or more instances of per-state scripts to be running (which will then each be polling the database and just start a transcode or cut or whatever script as needed), SReview uses a single "dispatch" script, which needs to be run once for the whole system (if using an external scheduler) or once per core that may be used (if not using an external scheduler), and which does all the database polling required. The use of an external scheduler seemed more appropriate, given that things like gridengine exist; gridengine is a job scheduler which allows one to submit a job to be ran on any node in a cluster, along with the resources that this particular job requires, and which will then either find an appropriate node to run the job on, or will put the job in a "pending" state until the required resources can be found. This allows me to more easily add extra encoding capacity when required, and allows me to also do things like allocate less resources to a particular part of the whole system, even while jobs are already running, without necessarily needing to abort jobs that might be using those resources.

The system seems to be working fine, although there's certainly still room for improvement. I'm thinking of using it for DebConf17 too, and will therefore probably work on improving it during DebCamp.

Additionally, the experience of using it for FOSDEM 2017 has given me ideas of where to improve it further, so it can be used more easily by different parties, too. Some of these have been filed as issues against a "1.0" milestone on github, but others are only newly formed in my gray matter and will need some thinking through before they can be properly implemented. Certainly, it looks like this will be something that'll give me quite some fun developing further.

In the mean time, if you're interested in the state of a particular video of FOSDEM 2017, have a look at the video overview page, which lists all talks along with their review/transcode status. Also, if you were a speaker or devroom organizer at FOSDEM 2017, please check your mailbox and review your talk! With your help, we should hopefully be able to release all our videos by the end of the week.

Update (2017-02-06 17:18): clarified my position on the qualities of some of the other systems after feedback from people who were a bit disappointed by my description of them... and which was fair enough. Apologies.

Jaldhar Vyas: Don't Believe Everything You Read on Debian Planet

6 February, 2017 - 12:41

Martin Pitt won the popular vote.

Russell Coker: SE Linux in Debian/Stretch

6 February, 2017 - 10:17

Debian/Stretch has been frozen. Before the freeze I got almost all the bugs in policy fixed, both bugs reported in the Debian BTS and bugs that I know about. This is going to be one of the best Debian releases for SE Linux ever.

Systemd with SE Linux is working nicely. The support isn’t as good as I would like, there is still work to be done for systemd-nspawn. But it’s close enough that anyone who needs to use it can use audit2allow to generate the extra rules needed. Systemd-nspawn is not used by default and it’s not something that a new Linux user is going to use, I think that expert users who are capable of using such features are capable of doing the extra work to get them going.

In terms of systemd-nspawn and some other rough edges, the issue is the difference between writing policy for a single system vs writing policy that works for everyone. If you write policy for your own system you can allow access for a corner case without a lot of effort. But if I wrote policy to allow access for every corner case then they might add up to a combination that can be exploited. I don’t recommend blindly adding the output of audit2allow to your local policy (be particularly wary of access to shadow_t and write access to etc_t, lib_t, etc). But OTOH if you have a system that’s running in enforcing mode that happens to have one daemon with more access than is ideal then all the other daemons will still be restricted.

As for previous releases I plan to keep releasing updates to policy packages in my own apt repository. I’m also considering releasing policy source to updates that can be applied on existing Stretch systems. So if you want to run the official Debian packages but need updates that came after Stretch then you can get them. Suggestions on how to distribute such policy source are welcome.

Please enjoy SE Linux on Stretch. It’s too late for most bug reports regarding Stretch as most of them won’t be sufficiently important to justify a Stretch update. The vast majority of SE Linux policy bugs are issues of denying wanted access not permitting unwanted access (so not a security issue) and can be easily fixed by local configuration, so it’s really difficult to make a case for an update to Stable. But feel free to send bug reports for Buster (Stretch+1).

Related posts:

  1. Debian SE Linux Status June 2012 It’s almost the Wheezy freeze time and I’ve been working...
  2. SE Linux Status in Debian 2012-01 Since my last SE Linux in Debian status report [1]...
  3. Debian SSH and SE Linux I have just filed Debian bug report #556644 against the...

Daniel Stender: Howto create a Debian 9 preview as Vagrant box with Packer

6 February, 2017 - 07:00

I’ve got some little scripts and a template here to automatically create Vagrant boxes from cutting edge Debian testing daily snapshots (netinstall ISO image) using HashiCorp’s Packer.

To create Vagrant boxes with these, you first need a running binary of Packer. There is a Debian package available if that’s also your working environment, but Packer is going to be introduced into the stable branch with the upcoming Stretch release itself. However, Ubuntu already has it, and some other derivatives, too. And there are prebuild binaries available from the developer’s site which run fine out-of-the-box (you just have to put the single binary somewhere into you $PATH, or expand that to find it). The JSON template should run with any Packer which is available for any of the different systems.

Vagrant itself isn’t needed to build the box with Packer, but Virtualbox is of course needed to pre bake the machine image within a virtual machine. In Debian the base binaries of Virtualbox are in the contrib archive section, so that source might be added to /etc/apt/sources.list, if haven’t already. The scripts have been tested to run with 5.1.10, and I haven’t seen that any late versions are demanded in particular, but of course heavily outdated versions might not work properly.

Packer installs the guest additions ISO file for Virtualbox into the virtual machine (and the shipped provisioning script then builds them inside). For that, the Debian package which ships that (which is in non-free) is recognized if it is installed, and then could be used by Packer. When the ISO isn’t available nowhere on the working machine the builder then automatically downloads the corresponding ISO from http://download.virtualbox.org/virtualbox.

When the tarball with the scripts is unpacked, just do make create and the process should run through, if Packer and Virtualbox are available. If your environment doesn’t have GNU Make nor wget you might want to copy the relevant lines from the Makefile and run it manually. But if it does, just do it like this:

/tmp/debian-testing-vagrantbox$ make create
virtualbox-iso output will be in this color.
==> virtualbox-iso: Downloading or copying Guest additions
    virtualbox-iso: Downloading or copying: file:///usr/share/virtualbox/VBoxGuestAdditions.iso
==> virtualbox-iso: Downloading or copying ISO
    virtualbox-iso: Downloading or copying: http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/debian-testing-amd64-netinst.iso
    virtualbox-iso: Download progress: 10%
{...}
    virtualbox-iso: Download progress: 96%
==> virtualbox-iso: Starting HTTP server on port 8219
==> virtualbox-iso: Creating virtual machine...
==> virtualbox-iso: Creating hard drive...
==> virtualbox-iso: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 2885)
==> virtualbox-iso: Starting the virtual machine...
==> virtualbox-iso: Waiting 10s for boot...
==> virtualbox-iso: Typing the boot command...
==> virtualbox-iso: Waiting for SSH to become available...

The Virtualbox window then pops up and the build process continues within the virtual machine for a while. You might want to file a Github issue when there’s a problem on your machine, please! (please include the tail of your packer.log)

The Packer template (debian-testing-vagrant.json) is described in the documentation of the virtualbox-iso builder. A preseeding script for the Debian Installer (preseed.cfg) is also included which gets injected into the virtual build environment during the build process. The creation progress of the Debian base installation could be easily monitored since the Virtualbox window is fully shown during the Packer run (if you “loose” your mouse pointer by clicking inside that window, do <Right>+<Control> to escape). For good performance, a fast internet connection is needed since a whole base system must be downloaded – if that’s available the whole automated process very only takes a couple of minutes to complete on a non-SSD machine.

When Packer has finished and a fresh box is created (the size is about 690 MB), it then could be used with Vagrant. Just add the new box with:

/tmp/debian-testing-vagrantbox$ vagrant box add stretch-preview debian-testing-vagrant.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'stretch-preview' (v0) for provider: 
    box: Unpacking necessary files from: file:///tmp/debian-testing-vagrantbox/debian-testing-vagrant.box
==> box: Successfully added box 'stretch-preview' (v0) for 'virtualbox'!

It then could be initialized within a random working directory with:

/tmp/myproject$ vagrant init stretch-preview
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

After that, you could launch the virtual box with:

/tmp/myproject$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'stretch-preview'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: myproject_default_1486321215067_75270
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => /tmp/myproject
/tmp/myproject$

Then you can SSH into it by doing (touch is used here only to point to the shared folder):

/tmp/myproject$ touch hello!
/tmp/myproject$ vagrant ssh -- -X

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
/usr/bin/xauth:  file /home/vagrant/.Xauthority does not exist
vagrant@packer-virtualbox-iso-1486319595:~$ $ cat /etc/debian_version
9.0
vagrant@packer-virtualbox-iso-1486319595:~$ ls /vagrant/
hello!  Vagrantfile
vagrant@packer-virtualbox-iso-1486319595:~$ exit
logout
Connection to 127.0.0.1 closed.
/tmp/myproject$ vagrant halt
==> default: Attempting graceful shutdown of VM...
/tmp/myproject$

If you haven’t worked with Vagrant before, maybe this is appealing. The experience differs from using a chroot. Packer makes it very convenient to keep freshly created boxes for it coming. Inside the box, to change the pre installed us keyboard layout just do sudo dpkg-reconfigure keyboard-configuration (no password needed for sudo), and then sudo systemctl restart keyboard-setup.service.

Have fun!

Iustin Pop: Short rant/review of La La Land

6 February, 2017 - 06:16

Warning: Spoilers below. Rant below. Much angry, MANY ALL-CAPS. You've been warned!

So, today we went to see "La La Land", because I've heard good things about it, and because I do enjoy good musicals. And because of this, I wrote this post, instead of what I originally had in mind (related to kernel configuration).

Was it a good movie? Definitely yes. Was it a good musical? So and so. Did I like the ending? HELL NO, over and over NO.

The movie itself was much better than I expected. I don't read plot details in advance nor real reviews, so I expected more of a musical, and less of a good plot. But the movie had a very good plot. Two young people, striving to fulfil their artistic dreams, fall in love, and they fight through-sometimes helping, sometimes hindering each other—until, finally, each gets their own breakthrough, etc.

The choice of actress was spot on—halfway through the movie, I was thinking that I can't imagine the same plot played by a different actress. Of course many other actresses could have played the part, but Emma Stone played so well, I have trouble seeing the same character with the same always half-happy, half-sad attitude. The choice of actor was I think OK—at first I was in doubt, but he played also well. Or maybe it was just that I couldn't identify with him at first. Not that I identify well with artists in general ☺

The dance scenes were OK, and the singing good, but as I said, the musical part was secondary to the actual struggles of the characters. The movie itself was, technically, very well done; a lot of filming was in bars/clubs/locations with difficult lighting, and the shooting was very good. They also had a scene on a pier, looking towards the ocean and the setting sun, and the characters walking towards the beach—so heavily back-lighted, and I kept thinking "If I get only one shot this perfectly exposed and colour correct(ed), I'm happy". So high notes here.

Back to the plot. The story of how she and him fought their own struggles was very nicely told. Tick-tack, up and down (hope and rejection), leaning on the other to get morale back, is a captivating story. The cliff-hanger at the pre-end with her career, the going back home, the last minute save, all very well told.

So at this stage, I would have given the movie a 9/10. And I was happy.

Then we have the usual "one character has to go away to a far away country for a long time", except in this case it was just 4 months. And they have the usual discussion "what do we do with our relation, where do we take it", and she says "I will always love you", to which he replies "And I will too" (or equivalent).

In my mind, this means they'll have to survive during the break, they'll have to also survive through his touring months/years, but in the end love will be stronger. Because this is what the movie told us until now, that she made it because of him, and he made it because of her. Neither of them would have been this strong without the other (he wouldn't have picked up the invitation from his old pal, she wouldn't have gone to the final audition request nor write the play which got her the audition/recognition). Estimated movie ending: awesome.

And then… something happens. The timeline jumps 5 years in the future (as expected), and she is famous, married (WITH SOMEONE ELSE) and happy mother of a 3-year old. Through fate, she and her husband enter the club of Sebastian (as he also fulfilled his dream), she and Sebastian see each other, he plays their song, during which we're served a re-run of the movie but in stupid "everything goes well" style (all bad events eliminated), in which it is she and Sebastian who enter the club (which belongs now to somebody else), and then we're back in real time, song ends, she and her husband leave, but before that she and Sebastian exchange one last smile, THE END.

And I'm sitting there, not believing my eyes. WHAT THE? So I get home, not write this post for four hours to calm down, but I can't. Because this doesn't make sense. AT ALL.

What does the internet say? Quoting from this CNN article, written exactly today. The director says:

"That ending was there from the get-go," [director Damien Chazelle] told CNN in a recent interview. "I think I just have a thing about love stories where the lovers don't wind up together at the end; I find it very romantic."

Huh, excuse me?

"I think there's a reason why most of the greatest love stories in history don't end with happily ever after," Chazelle said. "To me, if you're telling a story about love, love has to be bigger than the characters." Chazelle sees Mia and Sebastian's love as a "third character" and something that "lives on." "[The ending gives] you that sense that even if the relationship itself might be over in practical terms, the love is not over," he said. "The love lasts, and I think that's just a beautiful kind of thing."

OH FOR THE LOVE OF. This is a wishy-washy explanation that tries to approach the thing from the artistic side. No, this is bullshit, because of multiple things. Let me try to roll back and explain what I think was the intention.

  1. An earlier fight between Mia and Sebastian points to the fact that they're both very dedicated to their careers, and this means it's hard for them to stay together if they both chase their dream. He has to be on tour, and she has to rehearse for her play, so they won't see each other for at least two weeks (in this instance). Later, she calls him and leaves a message that she hasn't seen him in a while (complex scene which ends in another fight, which is very well done). So we see the conflict that seems to say "You can't have a relation of equals; one party has to give up their dream". While this might be partially true in the real world, I don't go to movies to see the real world.

  2. After the year-long window into their life, I can't think that either Sebastian or Mia can be really successful without the other; because they are so alike, so passionate about their dreams, that a normal person wouldn't be able to understand and push the other when they need. However, the ending show both Mia and Sebastian quite successful, so one has to wonder: did they make it alone? Sebastian seems so (we don't see a partner for him), Mia unclear, likely not. How did Sebastian get through? What did Mia find in her husband?

  3. This is very one-sided, since I'm a man, so bear with me: Sebastian helped Mia through her tough time. Once she got the breakthrough (and they split), she found somebody else, and I have to wonder in what circumstances they met. In the sense that maybe her husband only knew "successful Mia" and not "struggling/aspiring Mia". Her husband seems completely oblivious to all the eye contact between Mia and Sebastian in the club, seems to know Sebastian/about Sebastian not. How deep is their relation?

  4. This is still one sided, sorry. When they break up (before Mia leaves for Paris), Sebastian asks "so where do we go from here?". Mia says "Nowhere". He asks once more, she rejects him again. So after one year of mad love and cries and happy moments, he gives up over two sentences? He's been following his dream (proper Jazz) in spite of all downturns in life until then, but he gives up on his real love over this? It doesn't make sense; trying to identify my self with the character, I can't reconcile this scene at all, unless he didn't really love her.

So no, I don't see them ending apart as romantic. I see it as the director is saying "You can't have both love and your [career] dreams. Choose either.", and he gives the "love" fake ending in the mini-re-roll of the movie, and the "career" wrong ending in the actual ending. And worse, he does it by negating significant parts of the character development done until now.

Moreover, this conclusion is wrong. Wrong because this is a movie, and if movies don't manage to make you dream that you can achieve all, if movies tell you "choose either", then all is lost. Their love is not a separate character; them struggling to find each other in the successful phase of their life, learning to adapt to the new "he" and "she", would be the third thing. As it was shown, their love is simply a young love, that can't really survive the changes in life; they each said "I'll love you forever", but with this ending it sounds more "I'll cherish the memory of young you forever". Or differently said, it sounded like a cheap excuse to use when ending their relationship, in order to not negate the relationship itself.

My version of the movie is another half hour long. It explains how Sebastian get over the "only jazz is pure old jazz" and manages to build a successful business around his old-style-but-modern jazz, instead of the pop-style jazz of the touring band (while thinking about her). It explains how Mia becomes a successful actress and gets over her first/second movies (while thinking about him), because one movie doesn't make one really successful (that reminds me: 3 year old child after 5 year forward-jump? when/how did her career go?). Hell, make it even more bitter—show how their correspondence starts strong but becomes more and more sporadic over time, dying after the first 2 years. Show how both of them try other relations, and not find the same spark that they had before.

And then, after they have matured, they meet again. And, just like the first time, they fall for each other, once again. She for his music, him for her passion for acting/for acting itself. She finds that him naming his club after her suggestion is oh-so-grown-up-and-sweet, he is happy that she finally grew into what he saw in her from the beginning. And he sings their song once more.

But no. I'm not an artist, so I can only get the "die hope die die die love because I can" version. I still recommend the movie, but not the "after 5 years" scenes.

Also, I didn't get time to bike today nor yesterday, so all you really get here is an ANGRY RANT. Because while I drink the coffee black and the tea without sugar, I like my happy endings, DAMN IT.

Dirk Eddelbuettel: random 0.2.6

6 February, 2017 - 06:04

A pure maintenance release of the random package for truly (hardware-based) random numbers as provided by random.org is now on CRAN. As requested by CRAN, we made running tests optional. Not running tests is clearly one way of not getting (spurious, networking-related) failures ...

Courtesy of CRANberries comes a diffstat report for this release. Current and previous releases are available here as well as on CRAN.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้