Planet Debian

Subscribe to Planet Debian feed
Planet Debian - http://planet.debian.org/
Updated: 2 hours 28 min ago

Miriam Ruiz: Awesome Bullying Lesson

20 November, 2014 - 04:36

A teacher in New York was teaching her class about bullying and gave them the following exercise to perform. She had the children take a piece of paper and told them to crumple it up, stamp on it and really mess it up but do not rip it. Then she had them unfold the paper, smooth it out and look at how scarred and dirty is was. She then told them to tell it they’re sorry. Now even though they said they were sorry and tried to fix the paper, she pointed out all the scars they left behind. And that those scars will never go away no matter how hard they tried to fix it. That is what happens when a child bullies another child, they may say they’re sorry but the scars are there forever. The looks on the faces of the children in the classroom told her the message hit home.

( Source: http://www.buzzfeed.com/mjs538/awesome-bullying-lesson-from-a-new-york-teacher )

Erich Schubert: What the GR outcome means for the users

20 November, 2014 - 02:58
The GR outcome is: no GR necessary This is good news. Because it says: Debian will remain Debian, as it was the last 20 years. For 20 years, we have tried hard to build the "universal operating system", and give users a choice. We've often had alternative software in the archive. Debian has come up with various tool to manage alternatives over time, and for example allows you to switch the system-wide Java. You can still run Debian with sysvinit. There are plenty of Debian Developers which will fight for this to be possible in the future. The outcome of this resolution says:
  • Using a GR to force others is the wrong approach of getting compatibility.
  • We've offered choice before, and we trust our fellow developers to continue to work towards choice.
  • Write patches, not useless GRs. We're coders, not bureocrats.
  • We believe we can do this, without making it a formal MUST requirement. Or even a SHOULD requirement. Just do it.
The sysvinit proponents may perceive this decision as having "lost". But they just don't realize they won, too. Because the GR may easily have backfired on them. The GR was not "every package must support sysvinit". It was also "every sysvinit package must support systemd". Here is an example: eudev, a non-systemd fork of udev. It is not yet in Debian, but I'm fairly confident that someone will make a package of it after the release, for the next Debian. Given the text of the GR, this package might have been inappropriate for Debian, unless it also supports systemd. But systemd has it's own udev - there is no reason to force eudev to work with systemd, is there? Debian is about choice. This includes the choice to support different init systems as appropriate. Not accepting a proper patch that adds support for a different init would be perceived as a major bug, I'm assured. A GR doesn't ensure choice. It only is a hammer to annoy others. But it doesn't write the necessary code to actually ensure compatibility. If GNOME at some point decides that systemd as pid 1 is a must, the GR only would have left us three options: A) fork the previous version, B) remove GNOME, C) remove all other init systems. Does this add choice? No. Now, we can preserve choice: if GNOME decides to go systemd-pid1-only, we can both include a forked GNOME, and the new GNOME (depending on systemd, which is allowed without the GR). Or any other solution that someone codes and packages... Don't fear that systemd will magically become a must. Trust that the Debian Developers will continue what they have been doing the last 20 years. Trust that there are enough Debian Developers that don't run systemd. Because they do exist, and they'll file bugs where appropriate. Bugs and patches, that are the appropriate tools, not GRs (or trolling).

EvolvisForge blog: Valid UTF-8 but invalid XML

19 November, 2014 - 21:18

Another PSA: something surprising about XML.

As you might all know, XML must be valid UTF-8 (or UTF-16 (or another encoding supported by the parser, but one which yields valid Unicode codepoints when read and converted)). Some characters, such as the ampersand ‘&’, must be escaped (“&#38;” or “&#x26;”, although “&amp;” may also work, depending on the domain) or put into a CDATA section (“<![CDATA[&]]>”).

A bit surprisingly, a literal backspace character (ASCII 08h, Unicode U+0008) is not allowed in the text. I filed a bugreport against libxml2, asking it to please encode these characters.

A bit more research followed. Surprisingly, there are characters that are not valid in XML “documents” in any way, not even as entities or in CDATA sections. (xmlstarlet, by the way, errors out somewhat nicely for an unescaped literal or entity-escaped backspace, but behaves absolutely hilarious for a literal backspace in a CDATA section.) Basically, XML contains a whitelist for the following Unicode codepoints:

  • U+0009
  • U+000A
  • U+000D
  • U+0020‥U+D7FF
  • U+E000‥U+FFFD
  • U-00010000‥U-0010FFFF

Additionally, a certain number of codepoints is discouraged: U+007F‥U+0084 (IMHO wise), U+0086‥U+009F (also wise, but why allow U+0085?), U+FDD0‥U+FDEF (a bit surprisingly, but consistent with disallowing the backspace character), and the last two codepoints of every plane (U+FFFE and U+FFFF were already disallowed, but U-0001FFFE, U-0001FFFF, …, U-0010FFFF weren’t; this is extremely wise).

The suggestion seems to be to just strip these characters silently from the XML “document”.

I’m a bit miffed about this, as I don’t even use XML directly (I’m extending a PHP “webapplication” that is a SOAP client and talks to a Java™ SOAP-WS) and would expect this to preserve my strings, but, oh my. I’ve forwarded the suggestion to just strip them silently to the libxml2 maintainers in the aforementioned bug report, for now, and may even hack that myself (on customer-paid time). More robust than hacking the PHP thingy to strip them first, anyway – I’ve got no control over the XML after all.

Sharing this so that more people know that not all UTF-8 is valid in XML. Maybe it saves someone else some time. (Now wondering whether to address this in my xhtml_escape shell function. Probably should. Meh.)

Thorsten Glaser: Debian init system freedom of choice GR worst possible outcome

19 November, 2014 - 19:44

Apparently (the actual results have not yet been published by the Secretary), the GR is over, and the worst possible option has won. This is an absolutely ambiguous result, while at the same time sending a clear signal that Debian is not to be trusted wrt. investing anything into it, right now.

Why is this? Simply: “GR not required” means that “whatever people do is probably right”. Besides this, we have one statement from the CTTE (“systemd is default init system for jessie. Period.”) and nothing else. This means that runit, or upstart, or file-rc, or uselessd, can be the default init system for zurg^H^H^H^Hstretch, or even the only one. It also means that the vast majority of Debian Developers are sheeple, neither clearly voting to preserve freedom of choice between init systems for its users, nor clearly voting to unambiguously support systemd and progress over compatibility and choice, nor clearly stating that systemd is important but supporting other init systems is still recommended. (I’ll not go into detail on how the proposer of the apparently winning choice recommends others to ignore ftpmaster constraints and licences, and even suggests to run a GR to soften up the DFSG interpretation.) I’d have voted this as “no, absolutely not” if it was possible to do so more strongly.

Judging from the statistics, the only thing I voted above NOTA/FD is the one least accepted by DDs, although the only other proposal I considered is the first-rated of them: support for other init systems is recommended but not required. What made me vote it below NOTA/FD was: “The Debian Project makes no statement at this time on sysvinit support beyond the jessie release.” This sentence made even this proposal unbearable, unacceptable, for people wanting to invest (time, money, etc.) into Debian.

This opens up a very hard problem: I’m absolutely stunned by this and wondering what to do now. While there is no real alternative to Debian at $dayjob I can always create customised packages in my own APT repository, and – while it was great when those were eventually (3.1.17-1) accepted into Debian, even replacing the previous packages completely – it is simpler and quicker to not do so. While $dayjob benefits from having packages I work on inside Debian itself, even though I cannot always test all scenarios Debian users would need, some work reduction due to… reactions… already led to Debian losing out on Mediawiki for jessie and some additional suffering. With my own package repository, I can – modulo installing/debootstrap – serve my needs for $dayjob much quicker, easily, etc. and only miss out on absolutely delightful user feedback. But then, others could always package software I’m upstream of for Debian. Or, if I do not leave the project, continue doing so via QA uploads.

I’m also disappointed because I have invested quite some effort into trying to make Debian better (my idea to join as DD was “if I’ve got to use it, it better be damn good!”), into packaging software and convincing people at work that developing software as Debian packages instead of (or not) thinking of packaging later was good. I’ve converted our versions of FusionForge and d-push to Debian packages, and it works pretty damn well. Sometimes it needs backports of my own, but that’s the corportate world, and no problem to an experienced DD. (I just feel bad we lost some people, an FTP master along them, before this really gained traction.)

I’d convert to OpenBSD because, despite MirBSD’s history with them, they’re the only technically sound alternative, but apparently tedu (whom I respect technically, and who used to offer good advice to even me when asked, and who I think wouldn’t choose systemd himself) still (allying with the systemd “side” (I’m not against people being able to choose systemd, for the record, I just don’t want to be forced into it myself!)) has some sort of grudge against me. Plus, it’d be hard to get customers to follow. So, no alternative right now. But I’m used to managing my own forks of software; I’m doomed to basically hack and fix anything I use (I recently got someone who owns a licence to an old-enough Visual Studio version to transfer that to me, so I can hack on the Windows Mobile 6 version of Cachebox, to fix bugs in one of the geocaching applications I use. Now I “just” need to learn C# and the .NET Compact Framework. So I’m also used to some amount of pain.)

I’m still unresolved wrt. the attitude I should show the Debian project now. I had decided to just continue to live on, and work on the things I need done, but that was before this GR non-result. I absolutely cannot recommend anyone to “invest” into Debian (without sounding hypocriet), but I cannot recommend anything else either. I cannot justify leaving but don’t know if I want to stay. I think I should sleep over it.

One thing I promised, and thus will do, is to organise a meeting of the Debian/m68k people soonish. But then, major and important and powerful forces inside Debian still insist that Debian-Ports are not part of it… yet, all forks of Debian now suffer from the systemd adoption in it instead of having a freedom-of-choice upstream. I’ve said, and I still feel that systemd adoption should have done in a Debian downstream / (pure?) blend, and maybe (parts of) GNOME removed from Debian itself for it. (Adding cgroups support to the m68k kernel to support systemd was done. I adviced against it, on the grounds of memory and code size. But no downstream can remove it now.)

Rhonda D'Vine: The Pogues

19 November, 2014 - 18:10

Actually I was working already on a different music blog entry, but I want to get this one out. I was invited to join the Organic Dancefloor last thursday. And it was a really great experience. A lot of nice people enjoying a dance evening of sort of improvisational traditional folk dancing with influences from different parts of europe. Three bands playing throughout the evening. I definitely plan to go there again. :)

Which brings me to the band I want to present you now. They also play sort-of traditional songs, or at least with traditional instruments, and are also quite danceable to. This is about The Pogues. And these are the songs that I do enjoy listening to every now and then:

  • Medley: Don't meddle with the Medley. Rather dance to it.
  • Fairytale of New York: Well, we're almost in the season for it. :)
  • Streams of Whiskey: Also quite the style of song that they are known for and party with at concerts.

Like always, enjoy!

/music | permanent link | Comments: 0 | Flattr this

Jonathan Wiltshire: Getting things into Jessie (#4)

19 November, 2014 - 17:12
Make sure bug metadata is accurate

We use the metadata on the bugs you claim to have closed, as well as reading the bug report itself. You can help us out with severities, tags (e.g. blocks), and version information.

Don’t fall into the trap of believing that an unblock is a green light into Jessie. Britney still follows her validity rules, so if an RC bug appears to affect the unblocked version, it won’t migrate. Versions matter, not only the bug state (closed or open).

Getting things into Jessie (#4) is a post from: jwiltshire.org.uk | Flattr

Bastian Venthur: General Resolution is not required

19 November, 2014 - 15:21

The result for the General Resolution about the init system coupling is out and the result is, not quite surprisingly, “General Resolution is not required”.

When skimming over -devel or -private from time to time, one easily gets the impression that we are all a bunch of zealots, all too eager for fighting. People argue in the worst possible ways. People make bold statements about the future of Debian if solution X is preferred over Y. People call each other names. People leave the project.

At some point you realize, we’re not all a bunch of zealots, it is usually only the same small subset of people always involved in those discussions. It’s reassuring that we still seem to have a silent majority in Debian that, without much fuss, just do what they can to make Debian better. In this sense: A General Resolution is not required.

Dirk Eddelbuettel: R / Finance 2015 Call for Papers

19 November, 2014 - 07:56

Earlier today, Josh send the text below to the R-SIG-Finance list, and I updated the R/Finance website, including its Call for Papers page, accordingly.

We are once again very excised about our conference, thrilled about the four confirmed keynotes, and hope that many R / Finance users will not only join us in Chicago in May 2015 -- but also submit a exciting proposal.

So read on below, and see you in Chicago in May!

Call for Papers:

R/Finance 2015: Applied Finance with R
May 29 and 30, 2015
University of Illinois at Chicago, IL, USA

The seventh annual R/Finance conference for applied finance using R will be held on May 29 and 30, 2015 in Chicago, IL, USA at the University of Illinois at Chicago. The conference will cover topics including portfolio management, time series analysis, advanced risk tools, high-performance computing, market microstructure, and econometrics. All will be discussed within the context of using R as a primary tool for financial risk management, portfolio construction, and trading.

Over the past six years, R/Finance has included attendees from around the world. It has featured presentations from prominent academics and practitioners, and we anticipate another exciting line-up for 2015. This year will include invited keynote presentations by Emanuel Derman, Louis Marascio, Alexander McNeil, and Rishi Narang.

We invite you to submit complete papers in pdf format for consideration. We will also consider one-page abstracts (in txt or pdf format) although more complete papers are preferred. We welcome submissions for both full talks and abbreviated "lightning talks." Both academic and practitioner proposals related to R are encouraged.

All slides will be made publicly available at conference time. Presenters are strongly encouraged to provide working R code to accompany the slides. Data sets should also be made public for the purposes of reproducibility (though we realize this may be limited due to contracts with data vendors). Preference may be given to presenters who have released R packages.

The conference will award two (or more) $1000 prizes for best papers. A submission must be a full paper to be eligible for a best paper award. Extended abstracts, even if a full paper is provided by conference time, are not eligible for a best paper award. Financial assistance for travel and accommodation may be available to presenters, however requests must be made at the time of submission. Assistance will be granted at the discretion of the conference committee.

Please make your submission online at this link. The submission deadline is January 31, 2015. Submitters will be notified via email by February 28, 2015 of acceptance, presentation length, and financial assistance (if requested).

Additional details will be announced via the R/Finance conference website as they become available. Information on previous years' presenters and their presentations are also at the conference website.

For the program committee:

Gib Bassett, Peter Carl, Dirk Eddelbuettel, Brian Peterson, Dale Rosenthal,
Jeffrey Ryan, Joshua Ulrich

Simon McVittie: still aiming to be the universal operating system

19 November, 2014 - 07:00

Debian's latest round of angry mailing list threads have been about some combination of init systems, future direction and project governance. The details aren't particularly important here, and pretty much everything worthwhile in favour of or against each position has already been said several times, but I think this bit is important enough that it bears repeating: the reason I voted "we didn't need this General Resolution" ahead of the other options is that I hope we can continue to use our normal technical and decision-making processes to make Debian 8 the best possible OS distribution for everyone. That includes people who like systemd, people who dislike systemd, people who don't care either way and just want the OS to work, and everyone in between those extremes.

I think that works best when we do things, and least well when a lot of time and energy get diverted into talking about doing things. I've been trying to do my small part of the former by fixing some release-critical bugs so we can release Debian 8. Please join in, and remember to write good unblock requests so our hard-working release team can get through them in a finite time. I realise not everyone will agree with my idea of which bugs, which features and which combinations of packages are highest-priority; that's fine, there are plenty of bugs to go round!

Regarding init systems specifically, Debian 'jessie' currently works with at least systemd-sysv or sysvinit-core as pid 1 (probably also Upstart, but I haven't tried that) and I'm confident that Debian developers won't let either of those regress before it's released as Debian 8.

I expect the freeze for Debian 'stretch' (presumably Debian 9) to be a couple of years away, so it seems premature to say anything about what will or won't be supported there; that depends on what upstream developers do, and what Debian developers do, between now and then. What I can predict is that the components that get useful bug reports, active maintenance, thorough testing, careful review, and similar help from contributors will work better than the things that don't; so if you like a component and want it to be supported in Debian, you can help by, well, supporting it.

PS. If you want the Debian 8 installer to leave you running sysvinit as pid 1 after the first reboot, here's a suitable incantation to add to the kernel command-line in the installer's bootloader. This one certainly worked when KiBi asked for testing a few days ago:

preseed/late_command="in-target apt-get install -y sysvinit-core"

I think that corresponds to this line in a preseeding file, if you use those:

d-i preseed/late_command string in-target apt-get install -y sysvinit-core

A similar apt-get command, without the in-target prefix, should work on an installed system that already has systemd-sysv. Depending on other installed software, you might need to add systemd-shim to the command line too, but when I tried it, apt-get was able to work that out for itself.

If you use aptitude instead of apt-get, double-check what it will do before saying "yes" to this particular switchover: its heuristic for resolving conflicts seems to be rather more trigger-happy about removing packages than the one in apt-get.

Laura Arjona: Translating (reviewing) Debian package descriptions

19 November, 2014 - 06:22

Some days I feel super lazy but I still would like to go on contributing translations to Debian.
Then, I leave the web translations a bit, and change to translate or review Debian package descriptions.

It’s something that anybody can do without any knowledge of translation tools, since it is a very simple web interface, as you will see.

First you need to create a login account, then, login into the system.

And then, go to the page of your mother language (in my case, Spanish, “es”). You will see some introductory text, and the list of pending translations:

At the end of the page, there is the list of translations pending to review:

We should begin with this, so the work that other people already made arrives quickly its destination. And it’s the easiest part, as you will see. Let’s pick one of them (libvformat1-dev):


You see the short description in the original English, and the current translation (if there were changes from a former version, they are coloured too).

I didn’t know what the package libvformat1-dev does, but here’s a nice opportunity to learn aobut it a bit :)

The short description looks ok for me. Let’s go on to the long description:

It also looks correct for me. So I leave the text box as is, and go on until the bottom of the page:

and click “Accept as is”. That’s all!!

The system brings you back to the page with pending translations and reviews. Let’s pick another one: totem

I found a typo and corrected some other words, so I updated the text in the translation box, left a message to the other translators in the comment box, and clicked “Accept with changes”.

And… iterate.

When 3 translators agree in a translation, it becomes official, and its propagated to apt-cache, aptitude, synaptic, etc., and the website (packages.debian.org). This is the most difficult part (to get 3 reviews for each package description):  many language teams are small, and their workforce is spread in many fronts: translations for the website, news and announcements, debconf templates (the messages that are shown to the user when a package is installed), the Debian installer, the documentation, the package descriptions… So your help (even when you only review some translations from time to time) will be appreciated, for sure.


Filed under: Tools Tagged: Contributing to libre software, Debian, English, translations

Christian Perrier: Bug #770000

19 November, 2014 - 01:13
Martin Pitt reported Debian bug #770000 on Tuesday November 18th, against the release.debian.org pseudo-package.

Bug #760000 was reported as of August 30th: so there have been 10,000 bugs reported in 3 months minus 12 days. The bug rate increased quite significantly during the last weeks. We can suspect this is related to the release and the freeze (that triggers many unblock requests)

I find it interesting that this bug is directly related to the release, directly related to systemd and originated from one of the systemd packages maintainers, if I'm right.

So, I'll take this opportunity to publicly thank all people who have brought the systemd packages to what they are now, whether or not they're still maintaining the package. We've all witnessed that Debian if facing a strong social issue nowadays and I'm very deeply sad about this. I hope we'll be able to go through this without losing too many brilliant contributors, as it happened recently.

Please prove me right and do The Right Thing for me to be able to continue this silly "round bug number" contest and still believe that, some day, bug #1000000 will really happen and I'm still there to witness it.

Ah, and by the way, systemd bloody works on my system. I can't even remember when I switched to it. It Just Worked.

Michal &#268;iha&#345;: Mercurial support in Weblate

19 November, 2014 - 00:00

Weblate has started as a translation system tightly bound to Git version control system. This was in no means design decision, but rather it was the version control I've used. But this has shown not to be sufficient and other systems were requested as well. And Mercurial is first of them to be supported.

Weblate 2.0 already had separated VCS layer and adding another system to that is quite easy if you know the VCS you're adding. Unfortunately this wasn't the case for me with Mercurial as I've never used it for anything more serious than cloning a repository, committing fixes and pushing it back. Weblate needs a bit more than that, especially in regard to remote branches. But nevertheless I've figured out all operations and the implementation is ready in our Git.

In case somebody is interested in adding support for another version control, patches are always welcome!

Filed under: English phpMyAdmin SUSE Weblate | 0 comments | Flattr this!

Dirk Eddelbuettel: RcppAnnoy 0.0.3

18 November, 2014 - 18:48

Hours after the initial blog post announcing the first release of the new package RcppAnnoy, Qiang Kou sent us a very nice pull request adding mmap support in Windows.

So a new release with Windows support is on now CRAN, and Windows binaries should be available by this evening as usual.

To recap, RcppAnnoy wraps the small, fast, and lightweight C++ template header library Annoy written by Erik Bernhardsson for use at Spotify. RcppAnnoy uses Rcpp Modules to offer the exact same functionality as the Python module wrapped around Annoy.

Courtesy of CRANberries, there is also a diffstat report for this release. More detailed information is on the RcppAnnoy page page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Jonathan Wiltshire: Getting things into Jessie (#3)

18 November, 2014 - 18:16
Make sure everything you’ve changed is in the changelog

We do read the diffs in detail, and if there’s no explanation for something that’s changed we’ll ask. We also expect it to be in the changelog.

Do save some round-trips by making sure your changelog is in order. One round-trip about your package is an inconvenience; when it’s scaled up to the number of requests we receive, it’s a serious time-sink for us.

Getting things into Jessie (#3) is a post from: jwiltshire.org.uk | Flattr

Josselin Mouette: Introspection (not the GObject one)

18 November, 2014 - 17:00
Disclaimer: I’m not used to writing personal stuff on Debian channels. However, there is nothing new here for those who know me from other public channels.


Yesterday, I received the weirdest email from well-known troll MikeeUSA. He thought I shared his views of a horrible world full of bloodthirsty feminists using systemd in their quest for domination over poor white male heterosexuals. The most nauseating paragraph was probably the one where he showed signs of the mentality of a pedocriminal.

At first, I shrugged it off and sent him an email explaining I didn’t want anything with his stinky white male supremacist theories, assorted with a bit of taunting. But after discovering all that stuff was actually sent to public mailing lists, I took the time for a second look and started a bit of introspection.

MikeeUSA thought I was a white male supremacist because of the so-called SmellyWerewolf incident, 6 years ago.
Oh boy, people change in six years. Upon re-reading that, I had trouble admitting I was the one to write it. Memory is selective, and with time, you tend not to remember some gruesome details, especially the ones that conflict most with your moral values.

I can assure every reader that the only people I intended to mock then were those who mistook Debian mailing lists for advertising channels; but I understand now that my message must have caused pain to a lot more people than that. So, it may come late, but let me take this opportunity to offer my sincerest apologies to anyone I may have hurt at that time.


It may seem strange for someone with deeply-rooted values of equality to have written that. To have considered that it was okay to stereotype people. And I think I found this okay because to me, those people were given equal rights, and were therefore equal. But the fight for equality is not over when everyone is given the same rights. Not until they are given the same opportunities to exert those rights. Which does not happen when they live in a society that likes to fit them in little archetypal peg holes, never giving you the chance to question where those stereotypes come from.

For me, that chance came from an unusual direction: the fight against prostitution. This goes way back for me. Since when I was a teenager, I have always been ticked off at the idea of nonconsensual sex that somehow evades criminal responsibility because of money compensation. I never understood why it wasn’t considered as rape. Yet it sounded weird that a male heterosexual would hold such opinions; after all, male heterosexuals should go to prostitutes as a kind of social ritual, right?

It was only three years ago that an organization of men against prostitution was founded in France. Not only did I find out that I was not alone with my progressive ideas, I was given the opportunity to exchange with many men and women who had studied prostitution: its effects on victims, its relationship to rape culture and more generally to the place men and women hold in society. Because eventually, it all boils down to little peg holes in which we expect people to fit: the virile man or the faggot, the whore or the mother. For me, it was liberating. I could finally get rid of the discomfort of being a white male heterosexual that didn’t enter the little peg holes that were made for me.

And now, after Sweden 15 years ago, a new group of countries are finally adopting laws to criminalize the act of paying for sex. Including France. That’s too bad for MikeeUSA, but this country is no longer the eldorado for white male supremacists. And I’m proud that our lobbying made a contribution, however small, to that change.

Erich Schubert: Generate iptables rules via pyroman

18 November, 2014 - 15:46
Vincent Bernat blogged on using Netfilter rulesets, pointing out that inserting the rules one-by-one using iptables calls may leave your firewall temporarily incomplete, eventually half-working, and that this approach can be slow. He's right with that, but there are tools that do this properly. ;-) Some years ago, for a multi-homed firewall, I wrote a tool called Pyroman. Using rules specified either in Python or XML syntax, it generates a firewall ruleset for you. But it also adresses the points Vincent raised:
  • It uses iptables-restore to load the firewall more efficiently than by calling iptables a hundred times
  • It will backup the previous firewall, and roll-back on errors (or lack of confirmation, if you are remote and use --safe)
It also has a nice feature for the use in staging: it can generate firewall rule sets offline, to allow you reviewing them before use, or transfer them to a different host. Not all functionality is supported though (e.g. the Firewall.hostname constant usable in python conditionals will still be the name of the host you generate the rules on - you may want to add a --hostname parameter to pyroman) pyroman --print-verbose will generate a script readable by iptables-restore except for one problem: it contains both the rules for IPv4 and for IPv6, separated by #### IPv6 rules. It will also annotate the origin of the rule, for example:
# /etc/pyroman/02_icmpv6.py:82
-A rfc4890f -p icmpv6 --icmpv6-type 255 -j DROP
indicates that this particular line was produced due to line 82 in file /etc/pyroman/02_icmpv6.py. This makes debugging easier. In particular it allows pyroman to produce a meaningful error message if the rules are rejected by the kernel: it will tell you which line caused the rule that was rejected. For the next version, I will probably add --output-ipv4 and --output-ipv6 options to make this more convenient to use. So far, pyroman is meant to be used on the firewall itself. Note: if you have configured a firewall that you are happy with, you can always use iptables-save to dump the current firewall. But it will not preserve comments, obviously.

Jaldhar Vyas: And The Papers Want To Know Whose Shirts You Wear

18 November, 2014 - 13:52

Today I was walking past the Courant Institute at NYU when I saw a man wearing a t-shirt with a picture of a cow diagramming all the various cuts of beef.

Now I've lost all interest in science. Thanks a lot jerks.

Antoine Beaupré: bup vs attic silly benchmark

18 November, 2014 - 12:39

after see attic introduced in a discussion about bup, i figured out i could give it a try. it was answering two of my biggest concerns with bup:

  • backup removal
  • encryption

and seemed to magically out of nowhere and basically do everything i need, with an inline manual on top of it.

disclaimer

Note: this is not a real benchmark! i would probably need to port bup and attic to liw's seivot software to report on this properly (and that would amazing and really interesting, but it's late now). even worse, this was done on a production server with other stuff going on so take results with a grain of salt.

procedure and results

Here's what I did. I setup backups of my ridiculously huge ~/src directory on the external hard drive where I usually make my backups. I ran a clean backup with attic, than redid it, then I ran a similar backup with bup, then redid it. Here are the results:

anarcat@marcos:~$ sudo apt-get install attic # this installed 0.13 on debian jessie amd64
[...]
anarcat@marcos:~$ attic init /mnt/attic-test:
Initializing repository at "/media/anarcat/calyx/attic-test"
Encryption NOT enabled.
Use the "--encryption=passphrase|keyfile" to enable encryption.
anarcat@marcos:~$ time attic create --stats /mnt/attic-test::src ~/src/
Initializing cache...
------------------------------------------------------------------------------
Archive name: src
Archive fingerprint: 7bdcea8a101dc233d7c122e3f69e67e5b03dbb62596d0b70f5b0759d446d9ed0
Start time: Tue Nov 18 00:42:52 2014
End time: Tue Nov 18 00:54:00 2014
Duration: 11 minutes 8.26 seconds
Number of files: 283910

                       Original size      Compressed size    Deduplicated size
This archive:                6.74 GB              4.27 GB              2.99 GB
All archives:                6.74 GB              4.27 GB              2.99 GB
------------------------------------------------------------------------------
311.60user 68.28system 11:08.49elapsed 56%CPU (0avgtext+0avgdata 122824maxresident)k
15279400inputs+6788816outputs (0major+3258848minor)pagefaults 0swaps
anarcat@marcos:~$ time attic create --stats /mnt/attic-test::src-2014-11-18 ~/src/
------------------------------------------------------------------------------
Archive name: src-2014-11-18
Archive fingerprint: be840f1a49b1deb76aea1cb667d812511943cfb7fee67f0dddc57368bd61c4bf
Start time: Tue Nov 18 00:05:57 2014
End time: Tue Nov 18 00:06:35 2014
Duration: 38.15 seconds
Number of files: 283910

                       Original size      Compressed size    Deduplicated size
This archive:                6.74 GB              4.27 GB            116.63 kB
All archives:               13.47 GB              8.54 GB              3.00 GB
------------------------------------------------------------------------------
30.60user 4.66system 0:38.38elapsed 91%CPU (0avgtext+0avgdata 104688maxresident)k
18264inputs+258696outputs (0major+36892minor)pagefaults 0swaps
anarcat@marcos:~$ sudo apt-get install bup # this installed bup 0.25
anarcat@marcos:~$ free && sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && free # flush caches
anarcat@marcos:~$ export BUP_DIR=/mnt/bup-test
anarcat@marcos:~$ bup init
Dépôt Git vide initialisé dans /mnt/bup-test/
anarcat@marcos:~$ time bup index ~/src
Indexing: 345249, done.
56.57user 14.37system 1:45.29elapsed 67%CPU (0avgtext+0avgdata 85236maxresident)k
699920inputs+104624outputs (4major+25970minor)pagefaults 0swaps
anarcat@marcos:~$ time bup save -n src ~/src
Reading index: 345249, done.
bloom: creating from 1 file (200000 objects).
bloom: adding 1 file (200000 objects).
bloom: creating from 3 files (600000 objects).
Saving: 100.00% (6749592/6749592k, 345249/345249 files), done.
bloom: adding 1 file (126005 objects).
383.08user 61.37system 10:52.68elapsed 68%CPU (0avgtext+0avgdata 194256maxresident)k
14638104inputs+5944384outputs (50major+299868minor)pagefaults 0swaps
anarcat@marcos:attic$ time bup index ~/src
Indexing: 345249, done.
56.13user 13.08system 1:38.65elapsed 70%CPU (0avgtext+0avgdata 133848maxresident)k
806144inputs+104824outputs (137major+38463minor)pagefaults 0swaps
anarcat@marcos:attic$ time bup save -n src2 ~/src
Reading index: 1, done.
Saving: 100.00% (0/0k, 1/1 files), done.
bloom: adding 1 file (1 object).
0.22user 0.05system 0:00.66elapsed 42%CPU (0avgtext+0avgdata 17088maxresident)k
10088inputs+88outputs (39major+15194minor)pagefaults 0swaps

Disk usage is comparable:

anarcat@marcos:attic$ du -sc /mnt/*attic*
2943532K        /mnt/attic-test
2969544K        /mnt/bup-test

People are encouraged to try and reproduce those results, which should be fairly trivial.

Observations

Here are interesting things I noted while working with both tools:

  • attic is Python3: i could compile it, with dependencies, by doing apt-get build-dep attic and running setup.py - i could also install it with pip if i needed to (but i didn't)
  • bup is Python 2, and has a scary makefile
  • both have an init command that basically does almost nothing and takes little enough time that i'm ignoring it in the benchmarks
  • attic backups are a single command, bup requires me to know that i first want to index and then save, which is a little confusing
  • bup has nice progress information, especially during save (because when it loaded the index, it knew how much was remaining) - just because of that, bup "feels" faster
  • bup, however, lets me know about its deep internals (like now i know it uses a bloom filter) which is probably barely understandable by most people
  • on the contrary, attic gives me useful information about the size of my backups, including the size of the current increment
  • it is not possible to get that information from bup, even after the fact - you need to du before and after the backup
  • attic modifies the files access times when backing up, while bup is more careful (there's a pull request to fix this in attic, which is how i found out about this)
  • both backup systems seem to produce roughly the same data size from the same input
Summary

attic and bup are about equally fast. bup took 30 seconds less than attic to save the files, but that's not counting the 1m45s it took indexing them, so on the total run time, bup was actually slower. attic is also (almost) two times faster on the second run as well. but this could be within the margin of error of this very quick experiment, so my provisional verdict for now would be that they are about as fast.

bup may be more robust (for example it doesn't modify the atimes), but this has not been extensively tested and is more based with my familiarity with the "conservatism" of the bup team rather than actual tests.

considering all the features promised by attic, it makes for a really serious contender to the already amazing bup.

Next steps

The properly do this, we would need to:

  • include other software (thinking of Zbackup, Burp, ddar, obnam, rdiff-backup and duplicity)
  • bench attic with the noatime patch
  • bench dev attic vs dev bup
  • bench data removal
  • bench encryption
  • test data recovery
  • run multiple backup runs, on different datasets, on a cleaner environment
  • ideally, extend seivot to do all of that

Vincent Sanders: NetSurf Developer workshop IV

18 November, 2014 - 03:54
Over the weekend the NetSurf developers met to make a concentrated effort on improving the browser. This time we were kindly hosted by Codethink in their Manchester office in a pleasant environment with plenty of refreshments.

Five developers managed to attend in person from around the UK: Michael Drake, John-Mark Bell, Daniel Silverstone, Rob Kendrick and Vincent Sanders. We also had Chris Young providing some bug fixes remotely.

We started the weekend by discussing all the thorny core issues that had been put on the agenda and ensuring the outcomes were properly noted. We also held the society AGM which was minuted by Daniel.

The emphasis of this weekend was very much on planning and doing the disruptive changes we had been putting off until we were all together.

John-Mark and myself managed to change the core build system as used by all the libraries to using standard triplets to identify systems and use the gnu autoconf style of naming for parameters (i.e. HOST, BUILD and CC being used correctly).

This was accompanied by improvements and configuration changes to the CI system to accommodate the new usage.

Several issues from the bug tracker were addressed and we put ourselves in a stronger position to address numerous other usability problems in the future.

We managed to pack a great deal into the 20 hours of work on Saturday and Sunday although because we were concentrating much more on planning and infrastructure rather than a release the metrics of commits and files changed were lower than at previous events.

Niels Thykier: The first 12 days and 408 unblock requests into the Jessie freeze

18 November, 2014 - 03:17

The release team receives an extreme amount of unblock requests right now.  For the past 22 days[1], we have been receiving no less than 408 unblock/ageing requests.  That is an average of ~18.5/day.  In the same period, the release team have closed 350 unblocks requests, averaging 15.9/day.

This number does not account for number of unblocks, we add without a request, when we happen to spot when we look at the list of RC bugs[2]. Nor does it account for unblock requests currently tagged “moreinfo”, of which there are currently 25.

All in all, it has been 3 intensive weeks for the release team.  I am truly proud of my fellow team members for keeping up with this for so long!  Also a thanks to the non-RT members, who help us by triaging and reviewing the unblock requests!  It is much appreciated. :)

 

Random bonus info:

  • d (our diffing tool) finally got colordiff support during the Release Sprint last week.  Prior to that, we got black’n’white diffs!
    • ssh coccia.debian.org -t /srv/release.debian.org/tools/scripts/d <srcpkg>
    • Though coccia.debian.org do not have colordiff installed right now.  I have filed a request to have it installed.
  • The release team have about 132 (active) unblock hints deployed right now in our hint files.

 

[1] We started receiving some in the 10 days before the freeze as people realised that their uploads would need an unblock to make it into Jessie.

[2] Related topics: “what is adsb?” (the answer being: Our top hinter for Wheezy)

 


Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้