Planet Debian

Subscribe to Planet Debian feed
Planet Debian - https://planet.debian.org/
Updated: 38 min ago

Dirk Eddelbuettel: RcppArmadillo 0.8.400.0.0

2 March, 2018 - 06:19

RcppArmadillo release 0.8.400.0.0, originally prepared and uploaded on February 19, finally hit CRAN today (after having been available via the RcppCore drat repo for a number of days). A corresponding Debian release was prepared and uploaded as well. This RcppArmadillo release contains Armadillo release 8.400.0 with a number of nice changes (see below for details), and continues our normal bi-monthly CRAN release cycle (slight delayes in CRAN processing notwithstanding).

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 450 other packages on CRAN.

A high-level summary of changes follows.

Changes in RcppArmadillo version 0.8.400.0.0 (2018-02-19)
  • Upgraded to Armadillo release 8.400.rc2 (Entropy Bandit)

    • faster handling of sparse matrices by repmat()

    • faster loading of CSV files

    • expanded kron() to handle sparse matrices

    • expanded index_min() and index_max() to handle cubes

    • expanded randi(), randu(), randn(), randg() to output single scalars

    • added submatrix & subcube iterators

    • added normcdf()

    • added mvnrnd()

    • added chi2rnd()

    • added wishrnd() and iwishrnd()

  • The configure generated header settings for LAPACK and OpenMP can be overridden by the user.

  • This release was preceded by two release candidates which were tested extensively.

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Gregor Herrmann: trains & snow

2 March, 2018 - 05:08

last weekend I attended the Debian SnowCamp at the lago maggiore in north-western italy, a small DIY hacking & socialising Debian meeting in the tradion of Debian SunCamp. – some impressions:

travel

I wasn't aware of how tedious it can be to travel less than 500 kilometers into a neighbouring country by train in 2018. first of all, no train company would sell me a ticket from innsbruck to laveno-mombello, so innsbruck–verona from öbb & verona–laveno-mombello from trenitalia it was.

from the eight trains (yes, 3 changes per direction) exactly 0 had no delays at departure or arrival or both. in the end I "only" lost one connection. – & one direction from-door-to-door took roughly 9 hours.

location

the ostello casa rossa was lovely (& might be even more lovely with warmer temperature which would allow for longer stays in the garden), & the food at the ristorante concordia was mostly excellent. trying new local specialties (like the pizzoccheri) is always fun. & no lunch or dinner lasted shorter than 2 hours.

work

unsurprisingly, my work was mostly focussed on Debian Perl Group stuff. we managed to move our repos from alioth to salsa during the weekend, which involved not only importing ~3500 repositories but also e.g. recreating our .mrconfig setup.

in practice it didn't help alot for my contributions that I was at SnowCamp as the others were not there, & coordination happened via IRC (unlike a team sprint); but at least I had more people who listened to my shouts of joy or frustration than what I would have had at home :)

misc
  • on my way back I even had snow while waiting at the delayed train in the unexciting station of laveno-mombello.
  • mille grazie to elena & friends for the perfect organisation!
  • if you speak german: the respective volume of my archäologie des alltags (also contains links to tweets with photos).
  • I see quite some potential for other Debian *Camps, maybe even longer & with more pre-planned team sprints under one roof (or in one garden) …

Antoine Beaupré: February 2018 report: LTS, ...

2 March, 2018 - 00:08
Debian Long Term Support (LTS)

This is my monthly Debian LTS report. This month was exclusively dedicated to my frontdesk work. I actually forgot to do it the first week and had to play catchup during the weekend, so I brought up a discussion about how to avoid those problems in the future. I proposed an automated reminder system, but it turns out people found this was overkill. Instead, Chris Lamb suggested we simply send a ping to the next person in the list, which has proven useful the next time I was up. In the two weeks I was frontdesk, I ended up triaging the following notable packages:

  • isc-dhcp - remote code execution exploits - time to get rid of those root-level daemons?
  • simpleasmlphp - under embargo, quite curious
  • golang - the return of remote code execution in go get (CVE-2018-6574, similar to CVE-2017-15041 and CVE-2018-7187) - ended up being marked as minor, unfortunately
  • systemd - CVE-2017-18078 was marked as unimportant as this was neutralized by kernel hardening and systemd was not really in use back in wheezy. besides, CVE-2013-4392 was about a similar functionality which was claimed to not be supported in wheezy. i did, however, proposed to forcibly enable the kernel hardening through default sysctl configurations (Debian bug #889098) so that custom kernels would be covered by the protection in stable suites.

There were more minor triage work not mentioned here, those are just the juicy ones...

Speaking of juicy, the other thing I did during the month was to help with the documentation on the Meltdown and Spectre attacks on Intel CPUs. Much has been written about this and I won't do yet another summary. However, it seems that no one actually had written even semi-official documentation on the state of fixes in Debian, which lead to many questions to the (LTS) security team(s). Ola Lundqvist did a first draft of a page detailing the current status, and I expanded on the page to add formatting and more details. The page is visible here:

https://wiki.debian.org/DebianSecurity/SpectreMeltdown

I'm still not fully happy with the results: we're missing some userland like Qemu and a timeline of fixes. In comparison, the Ubuntu page still looks much better in my opinion. But it's leagues ahead of what we had before, which was nothing... The next step for LTS is to backport the retpoline fixes back into a compiler. Roberto C. Sanchez is working on this, and the remaining question is whether we try to backport to GCC 4.7 or we backport GCC 4.9 itself into wheezy. In any case, it's a significant challenge and I'm glad I'm not the one dealing with such arcane code right now...

Other free software work

Not much to say this month, en vrac:

  • did the usual linkchecker maintenance
  • finally got my Prometheus node exporter directory size sample merged
  • added some docs updating the Dat project comparison with IPFS after investigating Dat. Turns out Dat's security garantees aren't as good as I hoped...
  • reviewed some PRs in the Git-Mediawiki project
  • found what I consider to be a security issue in the Borg backup software, but was disregarded as such by upstream. This ended up in a simple issue that I do not hope much from.
  • so I got more interested in the Restic community as well. I proposed a code of conduct to test the waters, but the feedback so far has been mixed, unfortunately.
  • started working on a streams page for the Sigal gallery. Expect an article about Sigal soon.
  • published undertime in Debian, which brought a slew of bug reports (and consequent fixes).
  • started looking at alternative GUIs because GTK2 is going a way and I need to port two projects. I have a list of "hello world" in various frameworks now, still not sure which one I'll use.
  • also worked on updating the Charybdis and Atheme-services packages with new co-maintainers (hi!)
  • worked with Darktable to try and render an exotic image out of my new camera. Might turn into a LWN article eventually as well.
  • started getting more involved in the local free software forum, a nice little community. In particular, i went to a "repair cafe" and wrote a full report on the experience there.

I'm trying to write more for LWN these days so it's taking more time. I'm also trying to turn those reports into articles to help ramping up that rhythm, which means you'll need to subscribe to LWN to get the latest goods before the 2 weeks exclusivity period.

Russ Allbery: Review: Coding Freedom

1 March, 2018 - 11:56

Review: Coding Freedom, by E. Gabriella Coleman

Publisher: Princeton University Press Copyright: 2013 ISBN: 0-691-14461-3 Format: Trade paperback Pages: 223

Subtitled The Ethics and Aesthetics of Hacking, Coding Freedom is a rare beast in my personal reading: an academic anthropological study of a fairly new virtual community. It's possible that many books of this type are being written, but they're not within my normal reading focus. It's also a bit of an awkward review, since the community discussed here is (one of) mine. I'm going to have an insider's nitpicks and "well, but" reactions to the anthropology, which is a valid reaction but not necessarily the intended audience.

I'm also coming to this book about four years after everyone finished talking about it, and even longer after Coleman's field work in support of the book. I think Coding Freedom suffers from that lack of currency. If this book were written today, I suspect its focus would change, at least in part. More on that in a moment.

Coding Freedom's title is more subtle and layered than it may first appear. It is about the freedom to write code, and about free software as a movement, but not only that. It's also about how concepts of freedom are encoded in the culture and language of hacking communities, and about the concept of code as speech (specifically free speech in the liberal tradition). And the title also captures the idea of code switching, where a speaker switches between languages even in the middle of sentences. The free software community does something akin to code switching between the domains of technical software problems, legal problems, and political beliefs and ideologies. Coleman covers all of that ground in this book.

Apart from an introduction and conclusion, the book is divided into five chapters in three parts. The opening part talks about the typical life story and community involvement of a free software hacker and briefly sketches the legal history of free software licenses. The second part talks about the experience of hacking, with a particular focus on playful expression and the tension between collaboration, competitiveness, and proving one's membership in the group. The final part dives into software as speech, legal and political struggles against the DMCA and other attempts to restrict code through copyright law, and the free software challenge to the liberal regime of capitalism and private property, grounded in the also-liberal value of free speech.

There's a lot here to discuss, but it's also worth noting what's not here, and what I think would have been here if the same field work were done today. There's nothing about gender or inclusion, which have surpassed DMCA issues to become the political flash point de jour. (Coleman notes early in the book that she intentionally omitted that topic as one that deserves its own separate treatment.) The presentation of social norms and behaviors also felt strongly centered in an early 2000s attitude towards social testing, with low tolerance of people who haven't proven their competence. Coleman uses the term meritocracy with very few caveats and complications. I don't think one would do that in work starting today; the flaws, unwritten borders, and gatekeeping for who can participate in that supposed meritocracy are now more frequently discussed.

Those omissions left me somewhat uncomfortable throughout. Coleman follows the community self-image from a decade or more ago (which makes sense, given that's when most of her field research and the majority of examples she draws on in the book are from): valuing technical acumen and skilled play, devoted to free speech, and welcoming and valuing anyone with similar technical abilities. While this self-image is not entirely wrong, it hides a world of unspoken rules and vicious gatekeeping to control who gets to have free speech within the community, what types of people are valued, and who is allowed to not do emotional labor. And who isn't.

These are rather glaring gaps, and for me they limit the usefulness of Coding Freedom as an accurate analysis of the community.

That said, I do want to acknowledge that this wasn't Coleman's project. Her focus, instead, is on the way free software communities noticed and pushed into the open some hidden conflicts in the tradition of liberalism. Free political speech and democratic politics have gone hand-in-hand with capitalism and an overwhelming emphasis on private property, extended into purely virtual objects such as computer software. Free software questions that alliance, pokes at it, and at times even rips it apart.

The free software movement is deeply embedded in liberalism. Although it has members from anarchist, communist, and other political traditions, the general community is not very radical in its understanding of speech, labor, or politics. It has a long tradition of trying to avoid disruptive politics, apart from issues that touch directly on free software, to maximize its political alliances and avoid alienating any members. Free software is largely not a critique of liberalism from the outside; it's a movement that expresses a conflict inside the liberal tradition. It asks whether self-expression is consistent with, and more important than, private property, a question that liberalism otherwise attempts to ignore.

This is the part of the book I found fascinating: looking at my community from the outside, putting emergent political positions in that community into a broader context, and showing the complex and skillful ways that the community discusses, analyzes, and reaches consensus on those positions while retaining a broad base of support and growing membership. Coleman provides a sense of being part of something larger in the best and most complicated way: not a revolution, not an ideology, but a community with complex boundaries, rituals that are both scoffed at and followed, and gatekeeping behavior that exist in part because any human community will create and enforce boundaries.

When one is deeply inside a culture, it's easy to get lost in the ethical debates over whether a particular community behavior is good or bad. It takes an anthropologist to recast all those behaviors, good and bad, as humans being human, and to ask curious questions about what social functions those behaviors serve. Coding Freedom gave me a renewed appreciation of the insight that can come from the disinterested observer. If nothing else, it might help me choose my battles more strategically, and have more understanding and empathy.

This is a very academic work, at least compared to what I normally read. I never lost the thread of Coleman's argument, but I found it hard going and heavy on jargon in a few places. If, like me, you're not familiar with current work in anthropology, you'll probably feel like part of the discussion is going over your head, and that some terms you're reading with their normal English meaning are actually terms of art with more narrow and specific definitions. This is a book rather than an academic paper, and it does try to be approachable, but it's more research than popularization.

I wish Coding Freedom were more engaged with the problems of free software today, instead of the problems of free software in 2002, the era of United States v. Elcom Ltd. and Free Dmitry. I wish that Coleman had been far more critical of the concept of a meritocracy, and had dug deeper into the gatekeeping and boundaries around who is allowed to participate and who is discouraged or excluded. And while I'm not going to complain about academic rigor, I wish the prose were a bit lighter and a bit more approachable, and that it hadn't taken me months to read this book.

But, that said, I'm not sorry to have finally read it. The perspective from the anthropological view of one's own community is quite valuable. The distance provides an opportunity for less judgmental analysis, and a reminder that human social structures are robust and complex attempts to balance contradictory goals.

Coleman made me feel more connected, not to an overarching ideology or political goal, but to a tangled, flawed, dynamic, and responsive community, whose primary shared purpose is to support that human complexity. Sometimes it's easy to miss that forest for the day-to-day trees.

If you want to get more of a feel for Coleman's work, her keynote on Anonymous at DebConf14 in Portland in 2014 is very interesting and consistent in tone and approach with this book (albeit on a somewhat more controversial topic).

Rating: 6 out of 10

Paul Wise: FLOSS Activities February 2018

1 March, 2018 - 07:58
Changes Issues Review Administration
  • myrepos: merge patches, triage bugs
  • Debian: forward domain expiry, discuss sensitive files with service maintainer
  • Debian QA: bug triage
  • Debian package tracker: deploy latest code
  • Debian mentors: check why package wasn't uploaded, restart importer after crash
  • Debian wiki: remove extraneous tmp file, fix user email address, unblacklist IP addresses, whitelist email addresses, whitelist email domain
  • Debian website: investigate translation update issue
Communication Sponsors

The work on harmony and librecaptcha was sponsored by my employer. All other work was done on a volunteer basis.

Norbert Preining: Ten Mincho – Great font and ugly Adobe

1 March, 2018 - 06:18

I recently stumbled upon a very interesting article by Ken Lunde (well known from CJKV Information Processing book) on a new typeface for Japanese called Ten Mincho, designed by Ryoko Nishizuka and Robert Slimbach. Reading that the Kanji and Roman part is well balanced, and the later one designed by Robert Slimbach, I was very tempted to get these fonts for my own publications and reports. But well, not with Adobe

From the Ten Mincho Blog

The fonts are available at TypeKit, but a detailed study of the license terms and EULA gave me to cold shivers:

These are not perpetual licenses. You won’t have direct access to the font files, so you will need to keep the Creative Cloud application running in order to keep using the Marketplace fonts you’ve synced.
A few things to know about fonts from Marketplace

So may I repeat:

  • You pay for the fonts but you can only use them while running Creative Cloud.
  • You have no way to use the fonts with any other application.
  • Don’t even think about using the fonts on Linux with TeX.
  • We can remove these fonts at our free will from your library, that is not perpetual license.

Also when you purchase the fonts you are warned in the last step that:

All sales are final. Sorry, no refunds. Please contact us if you have any questions before purchasing.

So not only can you use the fonts you purchased freely, you also cannot ask for reimbursement.

Adobe, that is a shame.

John Goerzen: Emacs #2: Introducing org-mode

1 March, 2018 - 05:09

In my first post in my series on Emacs, I described returning to Emacs after over a decade of vim, and org-mode being the reason why.

I really am astounded at the usefulness, and simplicity, of org-mode. It is really a killer app.

So what exactly is org-mode?

I wrote yesterday:

It’s an information organization platform. Its website says “Your life in plain text: Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.”

That’s true, but doesn’t quite capture it. org-mode is a toolkit for you to organize things. It has reasonable out-of-the-box defaults, but it’s designed throughout for you to customize.

To highlight a few things:

  • Maintaining TODO lists: items can be scattered across org-mode files, contain attachments, have tags, deadlines, schedules. There is a convenient “agenda” view to show you what needs to be done. Items can repeat.
  • Authoring documents: org-mode has special features for generating HTML, LaTeX, slides (with LaTeX beamer), and all sorts of other formats. It also supports direct evaluation of code in-buffer and literate programming in virtually any Emacs-supported language. If you want to bend your mind on this stuff, read this article on literate devops. The entire Worg website
    is made with org-mode.
  • Keeping notes: yep, it can do that too. With full-text search, cross-referencing by file (as a wiki), by UUID, and even into other systems (into mu4e by Message-ID, into ERC logs, etc, etc.)

Getting started

I highly recommend watching Carsten Dominik’s excellent Google Talk on org-mode. It is an excellent introduction.

org-mode is included with Emacs, but you’ll often want a more recent version. Debian users can apt-get install org-mode, or it comes with the Emacs packaging system; M-x package-install RET org-mode RET may do it for you.

Now, you’ll probably want to start with the org-mode compact guide’s introduction section, noting in particular to set the keybindings mentioned in the activation section.

A good tutorial…

I’ve linked to a number of excellent tutorials and introductory items; this post is not going to serve as a tutorial. There are two good videos linked at the end of this post, in particular.

Some of my configuration

I’ll document some of my configuration here, and go into a bit of what it does. This isn’t necessarily because you’ll want to copy all of this verbatim — but just to give you a bit of an idea of some of what can be configured, an idea of what to look up in the manual, and maybe a reference for “now how do I do that?”

First, I set up Emacs to work in UTF-8 by default.

(prefer-coding-system 'utf-8)
(set-language-environment "UTF-8")

org-mode can follow URLs. By default, it opens in Firefox, but I use Chromium.

(setq browse-url-browser-function 'browse-url-chromium)

I set the basic key bindings as documented in the Guide, plus configure the M-RET behavior.

(global-set-key "\C-cl" 'org-store-link)
(global-set-key "\C-ca" 'org-agenda)
(global-set-key "\C-cc" 'org-capture)
(global-set-key "\C-cb" 'org-iswitchb)

(setq org-M-RET-may-split-line nil)

Configuration: Capturing

I can press C-c c from anywhere in Emacs. It will capture something for me, and include a link back to whatever I was working on.

You can define capture templates to set how this will work. I am going to keep two journal files for general notes about meetings, phone calls, etc. One for personal, one for work items. If I press C-c c j, then it will capture a personal item. The %a in all of these includes the link to where I was (or a link I had stored with C-c l).

(setq org-default-notes-file "~/org/tasks.org")
(setq org-capture-templates
      '(
        ("t" "Todo" entry (file+headline "inbox.org" "Tasks")
         "* TODO %?\n  %i\n  %u\n  %a")
        ("n" "Note/Data" entry (file+headline "inbox.org" "Notes/Data")
         "* %?   \n  %i\n  %u\n  %a")
        ("j" "Journal" entry (file+datetree "~/org/journal.org")
         "* %?\nEntered on %U\n %i\n %a")
        ("J" "Work-Journal" entry (file+datetree "~/org/wjournal.org")
         "* %?\nEntered on %U\n %i\n %a")
        ))
(setq org-irc-link-to-logs t)

I like to link by UUIDs, which lets me move things between files without breaking locations. This helps generate UUIDs when I ask Org to store a link target for future insertion.


(require 'org-id)
(setq org-id-link-to-org-use-id 'create-if-interactive)

Configuration: agenda views

I like my week to start on a Sunday, and for org to note the time when I mark something as done.


(setq org-log-done 'time)
(setq org-agenda-start-on-weekday 0)

Configuration: files and refiling

Here I tell it what files to use in the agenda, and to add a few more to the plain text search. I like to keep a general inbox (from which I can move, or “refile”, content), and then separate tasks, journal, and knowledge base for personal and work items.

  (setq org-agenda-files (list "~/org/inbox.org"
                               "~/org/email.org"
                               "~/org/tasks.org"
                               "~/org/wtasks.org"
                               "~/org/journal.org"
                               "~/org/wjournal.org"
                               "~/org/kb.org"
                               "~/org/wkb.org"
  ))
  (setq org-agenda-text-search-extra-files
        (list "~/org/someday.org"
              "~/org/config.org"
  ))

  (setq org-refile-targets '((nil :maxlevel . 2)
                             (org-agenda-files :maxlevel . 2)
                             ("~/org/someday.org" :maxlevel . 2)
                             ("~/org/templates.org" :maxlevel . 2)
                             )
        )
(setq org-outline-path-complete-in-steps nil)         ; Refile in a single go
(setq org-refile-use-outline-path 'file)

Configuration: Appearance

I like a pretty screen. After you’ve gotten used to org a bit, you might try this.

(require 'org-bullets)
(add-hook 'org-mode-hook
          (lambda ()
            (org-bullets-mode t)))
(setq org-ellipsis "⤵")

Coming up next…

This hopefully showed a few things that org-mode can do. Coming up next, I’ll cover how to customize TODO keywords and tags, archiving old tasks, forwarding emails to org-mode, and using git to synchronize between machines.

You can also see a list of all articles in this series.

Resources to accompany this article

Dirk Eddelbuettel: #17: Dependencies.

1 March, 2018 - 04:45

Dependencies are invitations for other people to break your package.
-- Josh Ulrich, private communication

Welcome to the seventeenth post in the relentlessly random R ravings series of posts, or R4 for short.

Dependencies. A truly loaded topic.

As R users, we are spoiled. Early in the history of R, Kurt Hornik and Friedrich Leisch built support for packages right into R, and started the Comprehensive R Archive Network (CRAN). And R and CRAN had a fantastic run with. Roughly twenty years later, we are looking at over 12,000 packages which can (generally) be installed with absolute ease and no suprises. No other (relevant) open source language has anything of comparable rigour and quality. This is a big deal.

And coding practices evolved and changed to play to this advantage. Packages are a near-unanimous recommendation, use of the install.packages() and update.packages() tooling is nearly universal, and most R users learned to their advantage to group code into interdependent packages. Obvious advantages are versioning and snap-shotting, attached documentation in the form of help pages and vignettes, unit testing, and of course continuous integration as a side effect of the package build system.

But the notion of 'oh, let me just build another package and add it to the pool of packages' can get carried away. A recent example I had was the work on the prrd package for parallel recursive dependency testing --- coincidentally, created entirely to allow for easier voluntary tests I do on reverse dependencies for the packages I maintain. It uses a job queue for which I relied on the liteq package by Gabor which does the job: enqueue jobs, and reliably dequeue them (also in a parallel fashion) and more. It looks light enough:

R> tools::package_dependencies(package="liteq", recursive=FALSE, db=AP)$liteq
[1] "assertthat" "DBI"        "rappdirs"   "RSQLite"   
R> 

Two dependencies because it uses an internal SQLite database, one for internal tooling and one for configuration.

All good then? Not so fast. The devil here is the very innocuous and versatile RSQLite package because when we look at fully recursive dependencies all hell breaks loose:

R> tools::package_dependencies(package="liteq", recursive=TRUE, db=AP)$liteq
 [1] "assertthat" "DBI"        "rappdirs"   "RSQLite"    "tools"     
 [6] "methods"    "bit64"      "blob"       "memoise"    "pkgconfig" 
[11] "Rcpp"       "BH"         "plogr"      "bit"        "utils"     
[16] "stats"      "tibble"     "digest"     "cli"        "crayon"    
[21] "pillar"     "rlang"      "grDevices"  "utf8"      
R>
R> tools::package_dependencies(package="RSQLite", recursive=TRUE, db=AP)$RSQLite
 [1] "bit64"      "blob"       "DBI"        "memoise"    "methods"   
 [6] "pkgconfig"  "Rcpp"       "BH"         "plogr"      "bit"       
[11] "utils"      "stats"      "tibble"     "digest"     "cli"       
[16] "crayon"     "pillar"     "rlang"      "assertthat" "grDevices" 
[21] "utf8"       "tools"     
R> 

Now we went from four to twenty-four, due to the twenty-two dependencies pulled in by RSQLite.

There, my dear friend, lies madness. The moment one of these packages breaks we get potential side effects. And this is no laughing matter. Here is a tweet from Kieran posted days before a book deadline of his when he was forced to roll a CRAN package back because it broke his entire setup. (The original tweet has by now been deleted; why people do that to their entire tweet histories is somewhat I fail to comprehened too; in any case the screenshot is from a private discussion I had with a few like-minded folks over slack.)

That illustrates the quote by Josh at the top. As I too have "production code" (well, CRANberries for one relies on it), I was interested to see if we could easily amend RSQLite. And yes, we can. A quick fork and few commits later, we have something we could call 'RSQLighter' as it reduces the dependencies quite a bit:

R> IP <- installed.packages()   # using my installed mod'ed version
R> tools::package_dependencies(package="RSQLite", recursive=TRUE, db=IP)$RSQLite
 [1] "bit64"     "DBI"       "methods"   "Rcpp"      "BH"        "bit"      
 [7] "utils"     "stats"     "grDevices" "graphics" 
R>

That is less than half. I have not proceeded with the fork because I do not believe in needlessly splitting codebases. But this could be a viable candidate for an alternate or shadow repository with more minimal and hence more robust dependencies. Or, as Josh calls, the tinyverse.

Another maddening aspect of dependencies is the ruthless application of what we could jokingly call Metcalf's Law: the likelihood of breakage does of course increase with the number edges in the dependency graph. A nice illustration is this post by Jenny trying to rationalize why one of the 87 (as of today) tidyverse packages has now state "ORPHANED" at CRAN:

An invitation for other people to break your code. Well put indeed. Or to put rocks up your path.

But things are not all that dire. Most folks appear to understand the issue, some even do something about it. The DBI and RMySQL packages have saner strict dependencies, maybe one day things will improve for RMariaDB and RSQLite too:

R> tools::package_dependencies(package=c("DBI", "RMySQL", "RMariaDB"), recursive=TRUE, db=AP)
$DBI
[1] "methods"

$RMySQL
[1] "DBI"     "methods"

$RMariaDB
 [1] "bit64"     "DBI"       "hms"       "methods"   "Rcpp"      "BH"       
 [7] "plogr"     "bit"       "utils"     "stats"     "pkgconfig" "rlang"    

R> 

And to be clear, I do not believe in giving up and using everything via docker, or virtualenvs, or packrat, or ... A well-honed dependency system is wonderful and the right resource to get code deployed and updated. But it required buy-in from everyone involved, and an understanding of the possible trade-offs. I think we can, and will, do better going forward.

Or else, there will always be the tinyverse ...

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Chris Lamb: Free software activities in February 2018

1 March, 2018 - 01:36

Here is my monthly update covering what I have been doing in the free software world in February 2018 (previous month):

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month I:



I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues:

  • Add support for comparing Berkeley DB files. (Unfortunately this is currently incomplete because the libraries do not report metadata reliably!) (#890528)
  • Add support for comparing "XMLBeans" binary schemas. [...]
  • Drop spurious debugging code in Android tests. [...]


Debian

My activities as the current Debian Project Leader are covered in my "Bits from the DPL" email to the debian-devel-announce mailing list.

Patches contributed
  • debian-policy: Replace dh_systemd_install with dh_installsystemd. (#889167)
  • juce: Missing build-depends on graphviz. (#890035)
  • roffit: debian/rules does not override targets as intended. (#889975)
  • bugs.debian.org: Please add rel="canonical" to bug pages. (#890338)
Debian LTS

This month I have been paid to work 18 hours on Debian Long Term Support (LTS). In that time I did the following:

Uploads
  • redis:
    • 4.0.8-1 — New upstream release and fix a potential hardlink vulnerability.
    • 4.0.8-2 — Also listen on ::1 (IPv6) by default. (#891432)
  • python-django:
    • 1.11.10-1 — New upstream security release.
    • 2.0.2-1 — New upstream security release.
  • redisearch:
    • 1.0.6-1 — New upstream release.
    • 1.0.7-1 — New upstream release & add Lintian overrides for package-does-not-install-examples.
    • 1.0.8-1 — New upstream release, which includes my reproducibility-related change improvement.
  • adminer:
    • 4.6.1-1 — New upstream release and override debian-watch-does-not-check-gpg-signature as upstream do not release signatures.
    • 4.6.2-1 — New upstream release.
  • process-cpp:
    • 3.0.1-3 — Make the documentation reproducible.
    • 3.0.1-4 — Correct Vcs-Bzr to Vcs-Git.
  • sleekxmpp (1.3.3-3) — Make the build reproducible. (#890193)
  • python-redis (2.10.6-2) — Correct autopkgtest dependencies and misc packaging updates.
  • bfs (1.2.1-1) — New upstream release.

I also made misc packaging updates for docbook-to-man (1:2.0.0-41), gunicorn (19.7.1-4), installation-birthday (8) & python-daiquiri (1.3.0-3).

Finally, I performed the following sponsored uploads: check-manifest (0.36-2), django-ipware (2.0.1-1), nose2 (0.7.3-3) & python-keyczar (0.716+ds-2).

Debian bugs filed
  • zsh: Please make apt install completion work on "local" files. (#891140)
  • git-gui: Ignores git hooks. (#891552)
  • python-coverage:
    • Installs pyfile.html into wrong directory breaking HTML report generation. (#890560)
    • Document copyright information for bundled JavaScript source. (#890578)
FTP Team

As a Debian FTP assistant I ACCEPTed 123 packages: apticron, aseba, atf-allwinner, bart-view, binutils, browserpass, bulk-media-downloader, ceph-deploy, colmap, core-specs-alpha-clojure, ctdconverter, debos, designate, editorconfig-core-py, essays1743, fis-gtm, flameshot, flex, fontmake, fonts-league-spartan, fonts-ubuntu, gcc-8, getdns, glyphslib, gnome-keyring, gnome-themes-extra, gnome-usage, golang-github-containerd-cgroups, golang-github-go-debos-fakemachine, golang-github-mattn-go-zglob, haskell-regex-tdfa-text, https-everywhere, ibm-3270, ignition-fuel-tools, impass, inetsim, jboss-bridger, jboss-threads, jsonrpc-glib, knot-resolver, libctl, liblouisutdml, libopenraw, libosmo-sccp, libtest-postgresql-perl, libtickit, linux, live-tasks, minidb, mithril, mutter, neuron, node-acorn-object-spread, node-babel, node-call-limit, node-color, node-colormin, node-console-group, node-consolidate, node-cosmiconfig, node-css-color-names, node-date-time, node-err-code, node-gulp-load-plugins, node-html-comment-regex, node-icss-utils, node-is-directory, node-mdn-data, node-mississippi, node-mutate-fs, node-node-localstorage, node-normalize-range, node-postcss-filter-plugins, node-postcss-load-options, node-postcss-load-plugins, node-postcss-minify-font-values, node-promise-retry, node-promzard, node-require-from-string, node-rollup, node-rollup-plugin-buble, node-ssri, node-validate-npm-package-name, node-vue-resource, ntpsec, nvidia-cuda-toolkit, nyx, pipsi, plasma-discover, pokemmo, pokemmo-installer, polymake, privacybadger, proxy-switcher, psautohint, purple-discord, pytest-astropy, pytest-doctestplus, pytest-openfiles, python-aiomeasures, python-coverage, python-fitbit, python-molotov, python-networkmanager, python-os-service-types, python-pluggy, python-stringtemplate3, python3-antlr3, qpack, quintuple, r-cran-animation, r-cran-clustergeneration, r-cran-phytools, re2, sat-templates, sfnt2woff-zopfli, sndio, thunar, uhd, undertime, usbauth-notifier, vmdb2 & xymonq.

I additionally filed 15 RC bugs against packages that had incomplete debian/copyright files against: browserpass, designate, fis-gtm, flex, gnome-keyring, ibm-3270, knot-resolver, libopenraw, libtest-postgresql-perl, mithril, mutter, ntpsec, plasma-discover, pytest-arraydiff & r-cran-animation.

Gunnar Wolf: Things that really matter

28 February, 2018 - 23:34


Jan Wagner: Deploying a (simple) docker container system

28 February, 2018 - 15:54

When a small platform for shipping containers is needed, not speaking about Kubernets or something, you have a couple of common things you might want to deploy at first.

Usual things that I have to roll out everytime deloying such a platform:

Bootstraping docker and docker-compose

Most services are build upon multiple containers. A useful tool for doing this is for example docker-compose where you can describe your whole 'application'. So we need to deploy it beside docker itself.

Deploying Watchtower

An essential operational part is to keep you container images up to date.

Watchtower is an application that will monitor your running Docker containers and watch for changes to the images that those containers were originally started from. If watchtower detects that an image has changed, it will automatically restart the container using the new image.

Deploying http(s) reverse proxy Træfik

If you want to provide multiple (web)services on port 80 and 443, you have to think about how this should be solved. Usually you would use a http(s) reverse proxy, there are many of software implementations available.
The challenging part in such an environment is that services may appear and disappear frequently. (Re)-configuration of the proxy service it the gap that needs to be closed.

Træfik (pronounced like traffic) is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease [...] to manage its configuration automatically and dynamically.

Træfik has many interesting features for example 'Let's Encrypt support (Automatic HTTPS with renewal)'.

John Goerzen: Emacs #1: Ditching a bunch of stuff and moving to Emacs and org-mode

28 February, 2018 - 05:40

I’ll admit it. After over a decade of vim, I’m hooked on Emacs.

I’ve long had this frustration over how to organize things. I’ve followed approaches like GTD and ZTD, but things like email or large files are really hard to organize.

I had been using Asana for tasks, Evernote for notes, Thunderbird for email, a combination of ikiwiki and some other items for a personal knowledge base, and various files in an archive directory on my PC. When my new job added Slack to the mix, that was finally the last straw.

A lot of todo-management tools integrate with email — poorly. When you want to do something like “remind me to reply to this in a week”, a lot of times that’s impossible because the tool doesn’t store the email in a fashion you can easily reply to. And that problem is even worse with Slack.

It was right around then that I stumbled onto Carsten Dominik’s Google Talk on org-mode. Carsten was the author of org-mode, and although the talk is 10 years old, it is still highly relevant.

I’d stumbled across org-mode before, but each time I didn’t really dig in because I had the reaction of “an outliner? But I need a todo list.” Turns out I was missing out. org-mode is all that.

Just what IS Emacs? And org-mode?

Emacs grew up as a text editor. It still is, and that heritage is definitely present throughout. But to say Emacs is an editor would be rather unfair.

Emacs is something more like a platform or a toolkit. Not only do you have source code to it, but the very configuration is a program, and there are hooks all over the place. It’s as if it was super easy to write a Firefox plugin. A couple lines, and boom, behavior changed.

org-mode is very similar. Yes, it’s an outliner, but that’s not really what it is. It’s an information organization platform. Its website says “Your life in plain text: Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.”

Capturing

If you’ve ever read productivity guides based on GTD, one of the things they stress is effortless capture of items. The idea is that when something pops into your head, get it down into a trusted system quickly so you can get on with what you were doing. org-mode has a capture system for just this. I can press C-c c from anywhere in Emacs, and up pops a spot to type my note. But, critically, automatically embedded in that note is a link back to what I was doing when I pressed C-c c. If I was editing a file, it’ll have a link back to that file and the line I was on. If I was viewing an email, it’ll link back to that email (by Message-Id, no less, so it finds it in any folder). Same for participating in a chat, or even viewing another org-mode entry.

So I can make a note that will remind me in a week to reply to a certain email, and when I click the link in that note, it’ll bring up the email in my mail reader — even if I subsequently archived it out of my inbox.

YES, this is what I was looking for!

The tool suite

Once you’re using org-mode, pretty soon you want to integrate everything with it. There are browser plugins for capturing things from the web. Multiple Emacs mail or news readers integrate with it. ERC (IRC client) does as well. So I found myself switching from Thunderbird and mairix+mutt (for the mail archives) to mu4e, and from xchat+slack to ERC.

And wouldn’t you know it, I liked each of those Emacs-based tools better than the standalone they replaced.

A small side tidbit: I’m using OfflineIMAP again! I even used it with GNUS way back when.

One Emacs process to rule them

I used to use Emacs extensively, way back. Back then, Emacs was a “large” program. (Now my battery status applet literally uses more RAM than Emacs). There was this problem of startup time back then, so there was a way to connect to a running Emacs process.

I like to spawn programs with Mod-p (an xmonad shortcut to a dzen menubar, but Alt-F2 in more traditional DEs would do the trick). It’s convenient to not run several emacsen with this setup, so you don’t run into issues with trying to capture to a file that’s open in another one. The solution is very simple: I created a script, named it em, and put it on my path. All it does is this:


#!/bin/bash
exec emacsclient -c -a "" "$@"

It creates a new emacs process if one doesn’t already exist; otherwise, it uses what you’ve got. A bonus here: parameters such as -nw work just fine, so it really acts just as if you’d typed emacs at the shell prompt. It’s a suitable setting for EDITOR.

Up next…

I’ll be talking about my use of, and showing off configurations for:

  • org-mode, including syncing between computers, capturing, agenda and todos, files, linking, keywords and tags, various exporting (slideshows), etc.
  • mu4e for email, including multiple accounts, bbdb integration
  • ERC for IRC and IM

Renata D'Avila: Woman. Not in tech.

28 February, 2018 - 04:00

Thank you, Livia Gabos, for helping me to improve this article by giving me feedback on it.

Before I became an intern with Outreachy, my Twitter bio read: "Woman. Not in tech." Well, if you didn't get the picture, let me explain that meant.

It all began with a simple request I received almost an year ago:

Hey, do you want to join our [company] event and give a talk about being a women in tech?

I don't have a job in the tech industry. So, yes, while society does put me in the 'woman' column, I have to admit it's a little hard to give a talk about being 'in tech' when I'm not 'in tech'.

What I can talk about, though, it's about all the women who are not in tech. The many, many friends I have who come to Women in Tech events and meetings, who reach out to me by e-mail, Twitter or even in person, who are struggling to get into tech.

I can talk about the only other girl in my class who, besides me, managed to get an internship. And how we both only got the position because we had passed a written exam about informatics, instead of going through usual channels such as referrals, CV analysis or interviews.

I can talk about the women who are seen as lazy, or that they just don't get it the lessons in tech courses because they don't have the same background and the same amount of time available to study or to do homework at home as their male peers do, since they have to take care of relatives, take care of children, take care of the housework for their family, most of the times while working in one or two jobs just to be able to study.

I can talk about the women and about the mothers who after many years being denied the possibility for a tech career are daring to change paths, but are denied junior positions in favor of younger men who "can be trained on the job" and have "so much more willingness to learn".

I can talk about the women who are seen as uninterested in one or more FLOSS technologies because they don't contribute to said technology, since the men in FLOSS projects have continuously failed in engage and - most importantly - keep them included (but maybe that's just because women lack role models).

Even though there are so many Women in Tech communities in Curitiba, as listed above, the all-male 'core team' of the local Debian community itself couldn't find a single woman to work with them for the DebConf proposal. Go figure.

I can talk about the many women I met not at tech conferences, but at teachers' conferences, that have way more experience with computers and programming than I. Women who after years working on the field have given up IT to become teachers, not because it was their lifelong dream, but because they didn't feel comfortable and well-integrated in a male-dominated full-of-mysoginistic-ideals tech industry. Because it was - and is - almost impossible for them to break the glass ceiling.

I can even talk about all the women who are lesbians that a certain community of Women In Tech could not find when they wanted someone to write an article about 'being homossexual in tech' to be published right on Brazil's Lesbian Visibility Day, so they had to go and ask a gay man to talk about his own experience. Well, it seems like those women aren't "in tech" either.

Tokenization can be especially apparent when the lone person in a minority group is not only asked to speak for the group, but is consistently asked to speak about being a member of that group. Geek Feminism - Tokenism

The things is, a lot of people don't want to hear any those stories. Companies in particular only want token women from outside the company (because, let's face it, most tech companies can't find the talent within) who will come up to the stage and inspire other women saying what a great experience it is to be in tech - and that "everyone should try it too!".

I do believe all women should try and get knowledge about tech and that is what I work towards. We shouldn't have to rely only on the men in our life to get things done with our computers or our cell phones or our digital life.

But to tell other women they should get into the tech industry? I guess not.

After all, who am I to tell other women they should come to tech - and to stay in tech - when I know we are bound to face all this?

Addendum:

For Brazilian women not in tech, I'm organizing a crowdfunding campaign to get at least five of them the opportunity to attend MiniDebConf in Curitiba, Parana, in April. None of these girls can afford the trip and they don't have a company to sponsor them. If you are willing to help, please get in touch or check this link: Women in MiniDebConf.

More on the subject:

Benjamin Mako Hill: XORcise

28 February, 2018 - 02:41

XORcise (ɛɡ.zɔʁ.siz) verb 1. To remove observations from a dataset if they satisfy one of two criteria, but not both. [e.g., After XORcising adults and citizens, only foreign children and adult citizens were left.]

Reproducible builds folks: Reproducible Builds: Weekly report #148

28 February, 2018 - 01:00

Here's what happened in the Reproducible Builds effort between Sunday February 18 and Saturday February 24 2018:

Logo and Outreachy/GSoC Reproducible work in other projects

There were a number of blog posts related to reproducible builds published this week:

Development and fixes in Debian key packages

Norbert Preining added calls to dh_stripnondeterminism to a number of TexLive packages which should let them become reproducible in Debian (#886988).

"Y2K-bug reloaded"

As part of the work on reproducible builds for openSUSE, Bernhard M. Wiedemann built packages 15 years in the future and discovered a widespread systematic errors in how Perl's Time::Local functions are used.

This affected a diverse set of software - including git and our strip-nondeterminism (via Archive::Zip)

grep was run on 16,896 tarballs in openSUSE's devel:languages:perl project and 102 of them contained timegm or timelocal calls. Of those, over 30 were problematic and some more need to be analyzed:

Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages

60 package reviews have been added, 32 have been updated and 30 have been removed in this week, adding to our knowledge about identified issues.

Two new toolchain issue types have been added:

Weekly QA work

During our reproducibility testing, FTBFS bugs have been detected and reported by:

  • Adrian Bunk (41)
  • Andreas Beckmann (1)
  • Boyuan Yang (1)
jenkins.debian.net development Misc.

This week's edition was written by Bernhard M. Wiedemann, kpcyrd, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Norbert Preining: CafeOBJ 1.5.7 released

27 February, 2018 - 22:40

Yesterday we have released CafeOBJ 1.5.7 with lots of changes concerning the inductive theorem prover CITP, as well as fixes to make CafeOBJ work with current SBCL. The documentation has gained a few more documents (albeit in Japanese), please see Documentation pages for the full list. The reference manual has been updated and is available as PDF, Html, or Wiki.

To quote from our README:

CafeOBJ is a new generation algebraic specification and programming language. As a direct successor of OBJ, it inherits all its features (flexible mix-fix syntax, powerful typing system with sub-types, and sophisticated module composition system featuring various kinds of imports, parameterised modules, views for instantiating the parameters, module expressions, etc.) but it also implements new paradigms such as rewriting logic and hidden algebra, as well as their combination.

Availability

Binary packages for Linux, MacOS, and Windows are already available, both in 32 and 64 bit and based on Allegro CL and SBCL (with some exceptions). All downloads can be found at the CafeOBJ download page. The source code can also be found on the download page, or directly from here: cafeobj-1.5.7.tar.gz.

The CafeOBJ Debian package is already updated.

Macports file has also been updated, please see the above download/install page for details how to add our sources to your macport.

Bug reports

If you find a bug, have suggestions, or complains, please open an issue at the Github issue page.

For other inquiries, please use info@cafeobj.org

Renata D'Avila: Working with git branches was the best decision I made

27 February, 2018 - 10:00

This is a short story about how chosing to use git branches saved me from some trouble.

How did I decide to use a new branch?

Up until certain point, I was just commiting all the code (and notes) I wrote into the master branch, which is the default for Git. No big deal, if I broke something, I could just go back and revert one or more commits and it would be okay.

It got to the point, though, that I would have send the code to people other than the mentors who had been seeing me breaking things: I would have to submit it for other developers to test it. While a broken code was the ideal for my mentors to see and help me in figuring out how to fix it, that wouldn't be useful for people seeing it in production and giving me feedback and suggestions for improvement. I had to send to them a good code that worked, and I had to do that while I worked on the last functionality needed, which is the recurrence rule for events.

After working on the recurrence rule for a few hours I realized that, since it wasn't really functional yet, I couldn't simply commit it on top of the rest of the code that had already been commited/published. Sure, I could have commented the function and commited the code that way, but it would be just too much trouble to have to comment/uncomment it every time I would work on that part.

That is how I chose to create a new git branch instead and starting commiting there the changes I had made.

I created a new branch called "devel" and asked git to change to it:

git checkout -b devel

Then, I did a "git status" to check everything was as expected and the branch had been changed to devel:

git status
renata@debian:~/foss_events$ git status
On branch devel

Next: staging the files and creating the commit:

git add --all .
git commit -m "COMMENTS FOR THE COMMIT"

Because the branch was created on my local machine, if I straight out try to just push the code upstream, it will give an error, because there is no "devel" branch on Github yet. So let's give some arguments to the git push command, asking it to set an upstream branch in the origin, which will receive the code:

git push --set-upstream origin devel
How did this save me from some trouble?

Simply because, before I sent the code to the moin-devel list, I decided to clean out the repository of the old and repeated code... by deleting those files. I wanted to do that so anyone who came to try it out would be able to spot the macro easily and not to worry whether the other files had to be installed anywhere for the code to work.

Once I had deleted those files using rm -r on the command line and right before I commited, I did a "git status" to check if the delete action had been recorded... that was when I noticed that I was still on the devel branch, and not on the master branch, where I wanted this change to take place! My development branch should stay the mess it was because it was stuff I used to try out.

I had used "rm -r", though, so how did I get those files back to the devel branch? I mean, the easy way, not the downloading-the-repo-again way.

Simple! I would have to discard the changes (deletes) I had made on the devel branch and change to the master branch to do it again:

BE CAREFUL, THIS COMMAND WILL DISCARD ANY CHANGES YOU HAVE MADE WHILE WORKING ON THE CURRENT BRANCH

git checkout -f

This will throw away the changes. Then, it's possible to move back to the master branch:

git checkout master

And, yup, I'm totally writing this article for future reference to myself.

Norbert Preining: Debian updates: CafeOBJ, Calibre, Html5-parser, LaTeXML, TeX Live

27 February, 2018 - 06:21

The last few days have seen a somehow quite unusual frenzy of uploads to Debian from my side. Mostly due to the fact that while doing my tax declaration (btw, a huge pain here in Japan) I needed some spare time and dedicated them to long overdue package maintenance work as well as some new request.

So what has happened (in alphabetic order):

  • CafeOBJ has been out of Debian/testing due to build errors on some platforms for quite some time. We (upstream) have fixed these incompatibilities which arose in minor version changes of SBCL, very disappointing. Anyway, we hope that the latest release of CafeOBJ will soon enter Debian/testing again.
  • Calibre gets the usual 2-3 weekly updates following upstream. Not much to report here – and in fact also not much work. There are a few items I still want to fix, in particular Rar support, but the maintainer of unrar, Martin Meredith is completely unresponsive, although I have submitted patches to reinstantiate the shared library. That means that CBR support etc is still missing from Debian’s Calibre.
  • Html5-parser is a support library for Calibre, which saw an update which I have finally packaged. I haven’t had any complications with the previous version, though.
  • LaTeXML hasn’t been updated in nearly 3 years in Debian, despite the fact that a new upstream is available for quite some time. I got contacted by upstream about this, and realized I had contact with the maintainer of LaTeXML back in 2015. He isn’t using and maintaining LaTeXML anymore, and kindly agreed that I take it over under the Debian TeX Maintainers umbrella. So I have updated the packaging for the new release.
  • TeX Live got the usual monthly update I reported about the other day.

I thought with all that done I can rest a bit and concentrate on my bread-job (software R&D engineer) or my sweets-job (Researcher in Mathematical Logic), but out of the blue a RC bug of the TeX Live packages just flew in. That will be another evening.

Enjoy.

Jo Shields: EOL notification – Debian 7, Ubuntu 12.04

27 February, 2018 - 01:31

Mono packages will no longer be built for these ancient distribution releases, starting from when we add Ubuntu 18.04 to the build matrix (likely early to mid April 2018).

Unless someone with a fat wallet screams, and throws a bunch of money at Azure, anyway.

Andrea Veri: Adding reCAPTCHA v2 support to Mailman

26 February, 2018 - 22:13

As a follow-up to the reCAPTCHA v1 post published back in 2014 here it comes an updated version for migrating your Mailman instance off from version 1 (being decommissioned on the 31th of March 2018) to version 2. The original python-recaptcha library was forked into https://github.com/redhat-infosec/python-recaptcha and made compatible with reCAPTCHA version 2.

The relevant changes against the original library can be resumed as follows:

  1. Added ‘version=2’ against displayhtml, load_scripts functions
  2. Introduce the v2submit (along with submit to keep backwards compatibility) function to support reCAPTCHA v2
  3. The updated library is backwards compatible with version 1 to avoid unexpected code breakages for instances still running version 1

The required changes are located on the following files:

/usr/lib/mailman/Mailman/Cgi/listinfo.py

--- listinfo.py	2018-02-26 14:56:48.000000000 +0000
+++ /usr/lib/mailman/Mailman/Cgi/listinfo.py	2018-02-26 14:08:34.000000000 +0000
@@ -31,6 +31,7 @@
 from Mailman import i18n
 from Mailman.htmlformat import *
 from Mailman.Logging.Syslog import syslog
+from recaptcha.client import captcha
 
 # Set up i18n
 _ = i18n._
@@ -244,6 +245,10 @@
     replacements[''] = mlist.FormatFormStart('listinfo')
     replacements[''] = mlist.FormatBox('fullname', size=30)
 
+    # Captcha
+    replacements[''] = captcha.displayhtml(mm_cfg.RECAPTCHA_PUBLIC_KEY, use_ssl=True, version=2)
+    replacements[''] = captcha.load_script(version=2)
+
     # Do the expansion.
     doc.AddItem(mlist.ParseTags('listinfo.html', replacements, lang))
     print doc.Format()

/usr/lib/mailman/Mailman/Cgi/subscribe.py

--- subscribe.py	2018-02-26 14:56:38.000000000 +0000
+++ /usr/lib/mailman/Mailman/Cgi/subscribe.py	2018-02-26 14:08:18.000000000 +0000
@@ -32,6 +32,7 @@
 from Mailman.UserDesc import UserDesc
 from Mailman.htmlformat import *
 from Mailman.Logging.Syslog import syslog
+from recaptcha.client import captcha
 
 SLASH = '/'
 ERRORSEP = '\n\n<p>'
@@ -165,6 +166,17 @@
             results.append(
     _('There was no hidden token in your submission or it was corrupted.'))
             results.append(_('You must GET the form before submitting it.'))
+
+    # recaptcha
+    captcha_response = captcha.v2submit(
+        cgidata.getvalue('g-recaptcha-response', ""),
+        mm_cfg.RECAPTCHA_PRIVATE_KEY,
+        remote,
+    )
+
+    if not captcha_response.is_valid:
+        results.append(_('Invalid captcha: %s' % captcha_response.error_code))
+
     # Was an attempt made to subscribe the list to itself?
     if email == mlist.GetListEmail():
         syslog('mischief', 'Attempt to self subscribe %s: %s', email, remote)

/usr/lib/mailman/templates/en/listinfo.html

--- listinfo.html	2018-02-26 15:02:34.000000000 +0000
+++ /usr/lib/mailman/templates/en/listinfo.html	2018-02-26 14:18:52.000000000 +0000
@@ -3,7 +3,7 @@
 <HTML>
   <HEAD>
     <TITLE><MM-List-Name> Info Page</TITLE>
-  
+    <MM-Recaptcha-Script> 
   </HEAD>
   <BODY BGCOLOR="#ffffff">
 
@@ -116,6 +116,11 @@
       </tr>
       <mm-digest-question-end>
       <tr>
+      <tr>
+        <td>Please fill out the following captcha</td>
+        <td><mm-recaptcha-javascript></TD>
+      </tr>
+      <tr>
 	<td colspan="3">
 	  <center><MM-Subscribe-Button></center>
     </td>

The updated RPMs are being rolled out to Fedora, EPEL 6 and EPEL 7. In the meantime you can find them here.

If Mailman complains about not being able to load recaptcha.client follow these steps:

cd /usr/lib/mailman/pythonlib
ln -s /usr/lib/python2.6/site-packages/recaptcha/client recaptcha

And then on {subscribe,listinfo}.py:

import recaptcha

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้