Planet Debian

Subscribe to Planet Debian feed
Planet Debian - https://planet.debian.org/
Updated: 14 min 40 sec ago

Matthew Garrett: Extending proprietary PC embedded controller firmware

18 November, 2019 - 15:19
I'm still playing with my X210, a device that just keeps coming up with new ways to teach me things. I'm now running Coreboot full time, so the majority of the runtime platform firmware is free software. Unfortunately, the firmware that's running on the embedded controller (a separate chip that's awake even when the rest of the system is asleep and which handles stuff like fan control, battery charging, transitioning into different power states and so on) is proprietary and the manufacturer of the chip won't release data sheets for it. This was disappointing, because the stock EC firmware is kind of annoying (there's no hysteresis on the fan control, so it hits a threshold, speeds up, drops below the threshold, turns off, and repeats every few seconds - also, a bunch of the Thinkpad hotkeys don't do anything) and it would be nice to be able to improve it.

A few months ago someone posted a bunch of fixes, a Ghidra project and a kernel patch that lets you overwrite the EC's code at runtime for purposes of experimentation. This seemed promising. Some amount of playing later and I'd produced a patch that generated keyboard scancodes for all the missing hotkeys, and I could then use udev to map those scancodes to the keycodes that the thinkpad_acpi driver would generate. I finally had a hotkey to tell me how much battery I had left.

But something else included in that post was a list of the GPIO mappings on the EC. A whole bunch of hardware on the board is connected to the EC in ways that allow it to control them, including things like disabling the backlight or switching the wifi card to airplane mode. Unfortunately the ACPI spec doesn't cover how to control GPIO lines attached to the embedded controller - the only real way we have to communicate is via a set of registers that the EC firmware interprets and does stuff with.

One of those registers in the vendor firmware for the X210 looked promising, with individual bits that looked like radio control. Unfortunately writing to them does nothing - the EC firmware simply stashes that write in an address and returns it on read without parsing the bits in any way. Doing anything more with them was going to involve modifying the embedded controller code.

Thankfully the EC has 64K of firmware and is only using about 40K of that, so there's plenty of room to add new code. The problem was generating the code in the first place and then getting it called. The EC is based on the CR16C architecture, which binutils supported until 10 days ago. To be fair it didn't appear to actually work, and binutils still has support for the more generic version of the CR16 family, so I built a cross assembler, wrote some assembly and came up with something that Ghidra was willing to parse except for one thing.

As mentioned previously, the existing firmware code responded to writes to this register by saving it to its RAM. My plan was to stick my new code in unused space at the end of the firmware, including code that duplicated the firmware's existing functionality. I could then replace the existing code that stored the register value with code that branched to my code, did whatever I wanted and then branched back to the original code. I hacked together some assembly that did the right thing in the most brute force way possible, but while Ghidra was happy with most of the code it wasn't happy with the instruction that branched from the original code to the new code, or the instruction at the end that returned to the original code. The branch instruction differs from a jump instruction in that it gives a relative offset rather than an absolute address, which means that branching to nearby code can be encoded in fewer bytes than going further. I was specifying the longest jump encoding possible in my assembly (that's what the :l means), but the linker was rewriting that to a shorter one. Ghidra was interpreting the shorter branch as a negative offset, and it wasn't clear to me whether this was a binutils bug or a Ghidra bug. I ended up just hacking that code out of binutils so it generated code that Ghidra was happy with and got on with life.

Writing values directly to that EC register showed that it worked, which meant I could add an ACPI device that exposed the functionality to the OS. My goal here is to produce a standard Coreboot radio control device that other Coreboot platforms can implement, and then just write a single driver that exposes it. I wrote one for Linux that seems to work.

In summary: closed-source code is more annoying to improve, but that doesn't mean it's impossible. Also, strange Russians on forums make everything easier.

comments

Joachim Breitner: Faster Winter 1: Vectors

18 November, 2019 - 07:02

(This is the first optimization presented in the “faster winter” series, please see that post for background information.)

Immutable vectors

The first very low hanging fruit was to be found in the Wasm.Runtime.Memory module. The memory of the Wasm module was defined as follows:

import Data.Vector (Vector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (Vector Word8)
  , _miMax :: Maybe Size
  }

An immutable vector in a mutable box (Mutable m is something like IORef or STRef, whatever is appropriate for the monad m).

This means that every write to the memory has to copy the whole memory into a new vector. This is nice from a semantic point of view (e.g. you can take a snapshot of the memory at a specific point in time, evaluate the code some more, and the immutable vector is untouched), but obviously very inefficient.

Mutable vectors

So I changed it to a mutable vector (MVector) that allows in-place updates:

import Data.Vector (MVector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (MVector (PrimState m) Word8)
  , _miMax :: Maybe Size
  }

I still need to place that mutable vector in a mutable box, because an MVector still has fixed size but Wasm programs can resize their memory. This mean we still copy the whole memory when the memory grows, but not during normal writes.

Improvement: Allocations: -97.71% Memory: +16.87% Time: -98.88% (Commit af6adfc...254d700)

This is much more … usable!

Oddly, memory consumption increases,and I am not entirely sure why: My guess is that because it runs faster, the garbage collector runs less often, the program can allocate more stuff in between garbage collector runs.

Unboxed vectors

In a second step I also replaced the vector with an unboxed vector: Above we have the normal Data.Vector vector, which stores pointers (8 bytes!) to elements, even if the element is a single byte like here. By importing Data.Vector.Unboxed (no other change!) I get a nicely packed array of bytes.

Interestingly, while it improved maximum memory usage quite a little bit, it looks like it made the program actually a bit slower – potentially because it now had to box and unbox these Word8 values.

Improvement: Memory: -27.04% Time: +15.82% (4061fe6)

There may be more improvements possible here (e.g. when writing a Word32, maybe that can be done in one go, instead of writing the four bytes separately), but I did not follow this clue any further for now.

Instead, the next thing that looked like it was worth investigating was overly polymorphic code.

Joachim Breitner: Faster Winter 1/7: Vectors

18 November, 2019 - 07:02

(This is the first optimization presented in the “faster winter” series, please see that post for background information.)

Immutable vectors

The first very low hanging fruit was to be found in the Wasm.Runtime.Memory module. The memory of the Wasm module was defined as follows:

import Data.Vector (Vector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (Vector Word8)
  , _miMax :: Maybe Size
  }

An immutable vector in a mutable box (Mutable m is something like IORef or STRef, whatever is appropriate for the monad m).

This means that every write to the memory has to copy the whole memory into a new vector. This is nice from a semantic point of view (e.g. you can take a snapshot of the memory at a specific point in time, evaluate the code some more, and the immutable vector is untouched), but obviously very inefficient.

Mutable vectors

So I changed it to a mutable vector (MVector) that allows in-place updates:

import Data.Vector (MVector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (MVector (PrimState m) Word8)
  , _miMax :: Maybe Size
  }

I still need to place that mutable vector in a mutable box, because an MVector still has fixed size but Wasm programs can resize their memory. This mean we still copy the whole memory when the memory grows, but not during normal writes.

Improvement: Allocations: -97.71% Memory: +16.87% Time: -98.88% (Commit af6adfc...254d700)

This is much more … usable!

Oddly, memory consumption increases,and I am not entirely sure why: My guess is that because it runs faster, the garbage collector runs less often, the program can allocate more stuff in between garbage collector runs.

Unboxed vectors

In a second step I also replaced the vector with an unboxed vector: Above we have the normal Data.Vector vector, which stores pointers (8 bytes!) to elements, even if the element is a single byte like here. By importing Data.Vector.Unboxed (no other change!) I get a nicely packed array of bytes.

Interestingly, while it improved maximum memory usage quite a little bit, it looks like it made the program actually a bit slower – potentially because it now had to box and unbox these Word8 values.

Improvement: Memory: -27.04% Time: +15.82% (4061fe6)

There may be more improvements possible here (e.g. when writing a Word32, maybe that can be done in one go, instead of writing the four bytes separately, but I did not follow this any further.

Instead, it was time to hunt down the space leak, as the interpreter should not need any significant amounts of memory, and instead execute the program using a constant and low amount of memory.

Shirish Agarwal: Unbelievable

18 November, 2019 - 03:26

This will be short as I have been having a touch of cold and flu over the last week. As I shared, I moved to mastodon from twitter and quite a number of people have moved on mastodon and many other instance which run mastodon. The best thing IMHO was that –
a. I came to know the wide range of projects that fediverse is attacking or working on. While I’m not going to dwelve much into fediverse in this blog post, maybe some future blog post either here or on itsfoss.com might be used to share some of the features which I thought were far better than twitter and the possibility of people challenging the status quo, especially since if you have a bit of experience servicing a web-server. It is very LAMPisque in nature, except of course they use postgres ( much better choice and probably much more scalable) although they are still trying to fix and figure out response times and such, being a non-corporate, non-ad service, they rely on people’s donations.

Unbelievable

I just started seeing Unbelievable which was produced and shown on netflix. While I probably need a week or two and would be staggering my viewing so that it doesn’t affect me emotionally as have been told by friends. I just viewed the first twenty minutes of the 1st episode and there itself, it seemed lot of things had gone wrong. Some of my observations –

a. What is not told is when the first call to 911 went, if it was told that the assault involved a young woman shouldn’t a lady officer have been sent to the scene to investigate. While I have heard that the cop defended the girl, perhaps it would have been better if a lady cop would have gone to the scene. She might have been a tad more empathatic and it probably would have been easier for the victim, in this case a young girl to share her experience.

b. When it was told it was rape, there were again two male detectives. Shouldn’t it be women handling this. I would have seen multiple advantages in having a woman detective here. Probably most women are able to have a good reading about other women. Probably women in law-enforcement much more so. She would also have been empathatic and usually girl/women are usually much more relaxed around other women. So for the detective it would also have helped her in the investigation for signs of ‘tells’ to know if the victim was lying.

c. No legal help for the victim – After the lead detective questions the girl again in the police station, he asks her to sign over the results of the tests of the testing from the lab as well as to write the statement again. Having an attorney there perhaps would have advised her as to what are her rights and whether she should or not either sign those documents or start writing by herself right in the police station which was stressful for her. As she does tell in the beginning of the interview, she has a headache. If such a statement was indeed needed by her, she should have been told to write in the confines of her home or where she was staying with her guardian. I know of so many times while I think of writing a blog post, there are so many ideas, when I start typing, half of them I simply have forgotten. Later, if I do even want to make a change or two, many a times the change itself would be so dramatic that I would have to re-write quite a bit of the blog post taking that additional info. which I had known but not voiced on the blog. So, when somebody like me who may miss formulations or ideas even when I’m not stressed, here is a young girl/woman who claims to have been raped/assaulted. Having had a lawyer even if she was not very sharp would probably told her of her rights and what she can or should not do in the cirumstances she is in. I probably will expand on it as I see more of the series either here itself or on other blog posts as I see more of it.

Hopefully we will meet in a more cheerful blog post

Joachim Breitner: Winter is coming even more quickly

18 November, 2019 - 00:06

TL;DR: I explain how I improved the performance of an interpreter for WebAssembly written in Haskell by plucking some low-hanging fruit.

Background

Motivated by my work at the [DFINITY Foundation], I was looking into interpreters for [WebAssembly] written in Haskell, and found my colleagues John Wiegley’s [winter]: A straight-forward port of the [WebAssembly reference interpreter], written in Ocaml by Andreas Rossberg (another colleague of mine … I guess there is a common theme here.)

Obviously, an interpreter will never be as fast as a real compiler implementation (such as the one in v8, lucet or wasmtime). But for my purposes an interpreter is fine. Nevertheless, I don’t want it to be needlessly slow, and when I picked up wasm, it was clear that I had to work at least a little bit on performance.

This is not to blame John, of course: I have no doubt that we could have sped it up at least as well! I assume it just wasn't a priority at that time.

A graph

I like statistics and graphs! In this blog post, I am always running the following very simple C program:

c unsigned int i; void init () { i = 1024*1024; while (i--) {}; }hs which I compile using clang version 9 and run with wasm-invoke from the winter package (compiled with GHC-8.6.5):

$ clang-9 --target=wasm32-unknown-unknown-wasm -fno-builtin -ffreestanding loop.c --compile
$ wasm-ld-9 loop.o --gc-sections --export init --no-entry -o loop.wasm
$ wasm-invoke -w loop.wasm -f init +RTS -t

The +RTS t tells the Haskell runtime to produce a line with some statistics, of which I collect

  • total number of byte allocated
  • maximum memory size used and
  • total time elapsed

To reduce the noise a little bit, I run the program five times, and take the minimum of all the runs.

I did this for all the commits I produce while trying to make the code faster. This yields the following graph:

The fruit of my labor

Note that I had to draw this with a logarithmic y axis, because some of the improvements were so drastic that a linear graph would be unhelpful.

But how did I get there? I will take you through a guided tour through all the commits that have improved performance, explain how I found an performance issue, what I did to fix I fixed it, and that improved things. To keep this post at a reasonable size, I split it into a series of blog post, faithful to the chronological order in which I performed them:

Here they are (will be published once per day):

  • Vectors
  • SPECIALIZE
  • Difference lists
  • Export lists
  • Eta-Expanding ReaderT
  • Simpler code
  • A Zipper
  • Statistics (the making off)

Was it worth it? I’d say so! Initially, running the program above would take 44 minutes (slow!) and continuously grows to require 6,7 GB of memory (Houston, we have a space leak!). After all the steps above, memory consumption is constant and below 2MB, and the program finishes in 3,6s. Overall, this yields the following, very satisfying, statistics:

Improvement: Allocations: -99.60% Memory: -99.97% Time: -99.86% (Commit de9f4f8...5406efd)

Wouter Verhelst: Announcing extrepo

17 November, 2019 - 16:34

Debian ships with a lot of packages. This allows our users to easily install software without too much effort -- just run apt-get install foo, and foo gets installed.

However, Debian does not ship with everything, and for that reason there sometimes are things that are not installable with just the Debian repositories. Examples include:

  • Software that is not (yet) packaged for Debian
  • A repository for software that is in Debian, but for the bleeding-edge version of that software (e.g., maintained by upstream, or maintained by the Debian packager for that software)
  • Software that is not DFSG-free, and cannot be included into Debian's non-free repository due to licensing issues, but that can freely be installed by Debian users.

In order to enable and use such repositories on a Debian system, a user currently has to perform some actions that may be insecure:

  • Some repositories provide a link to a repository and a key, and expect the user to perform all actions to enable that repository manually. This works, but mistakes are easy to make (especially for beginners), and therefore it is not ideal.
  • Some repositories provide a script to enable the repository, which must be run as root. Such scripts are not signed when run. In other words, we expect users to run untrusted code downloaded from the Internet, when everywhere else we tell people that doing so is a bad idea.
  • Some repositories provide a single .deb file that can be installed, and which enables the necessary repositories in the apt configuration. Since, in contrast to RPM files, Debian packages are not signed, this means that the configuration is not enabled securely.

While there is a tool to enable package signatures in Debian packages, the dpkg tool does not enforce the existence of such signatures, and therefore it is possible for an attacker to replace the (signed) .deb file with an unsigned variant, bypassing the whole signature.

In an effort to remedy this whole situation, I looked at creating extrepo, a package that would download repository metadata from a special-purpose repository, verify the signatures placed on that metadata, and if everything matches, enable the repository by creating the necessary apt configuration files.

This should allow users to enable external repository "foo" by running extrepo enable foo, rather than downloading a script from foo's website and executing it as root -- or other similarly insecure options.

The extrepo package has been uploaded to Debian; and so once NEW processing has finished, will be available in Debian unstable.

I might upload it to backports too if no issues are found, but that won't be an immediate thing.

Russ Allbery: Review: Rift in the Sky

17 November, 2019 - 13:25

Review: Rift in the Sky, by Julie E. Czerneda

Series: Stratification #3 Publisher: DAW Copyright: July 2009 ISBN: 1-101-13317-1 Format: Kindle Pages: 419

This is the third and final book of the Stratification trilogy and the bridge between it and the Trade Pact trilogy, of which it is a prequel. It's readable without reading the Trade Pact (although that series is overall better), but not very readable without the rest of the Stratification series.

Be warned that the publisher's summary of this book, found on Goodreads, Amazon, and other places you might read about it, is a spoiler for nearly all of the plot. It's not a very surprising spoiler if you've read the Trade Pact books, but still, publishers, don't do this.

Riders of the Storm left off with Aryl with a more stable situation, a new negotiated compromise with the Oud, and more information about the Om'ray, including the mysterious distant clan that no one visits. The origins of her world's balance of powers, and the goals of the outsider presence on her world, are both still opaque, but she's settled near the human Marcus and his archaeological site, so I was optimistic that we were about to learn more.

We don't. What the reader gets instead is more clan politics, worries about the new Om'ray powers that are eagerly adopted by the children of Aryl's clan, and only a few faint additional hints at the nature of the Cloisters. Aryl has more interactions with the Tiktik that finally lead to understanding more of the Agreement from the perspective of the other two races of Cersi (with rather dramatic consequences), and we learn quite a bit more about Om'ray reproduction. But there's next to nothing about how any of this strange and clearly artificial balance was created in the first place, or what the civilization that Marcus is so fascinated by truly is, or how it relates to the Om'ray, or even why the Om'ray clearly had advanced technology at one point in time that they no longer understand.

I hope the information that I wanted from this series is in the following Reunification series, since right now I'm irritated. (Although apparently not irritated enough to stop reading.)

On top of the frustrating lack of answers to nearly every question I had from the first book, this novel has a very odd structure, exacerbated by some strange decisions in how the Kindle version is configured. The last 100 pages are marked in the table of contents as "Teaser chapter" and appear after the dramatis personae. The Kindle reader even pops up the "rate this book" screen before that chapter, exactly as if the novel were over and this material were preview chapters of some subsequent book. I assumed it was a (surprisingly long) excerpt from the start of This Gulf of Time and Stars, the first book of the next trilogy.

It's not, although I admit I bought that book just to check for this review (I was going to buy it eventually anyway). That's a very good thing, since that last hundred pages was the only thing that salvaged this story for me, even though it uses my least favorite fictional trope.

The conclusion of the main story is weirdly abrupt. After lots of Aryl navigating the leadership of her clan, there's a very abrupt political shift (for magical reasons that I never found adequately explained) and the intensity of the pace picks up dramatically. Following some developments in the subplot with Marcus and some dramatic changes in the Agreement, Aryl and her people take drastic steps to resolve the situation, leading to the event that series readers will have been anticipating for the trilogy. But it's sudden and entirely unsatisfying, and if the book had actually ended there, this review would be much more negative.

Instead, the last part of the book takes us back into the science fiction setting of the previous trilogy and a reasonably entertaining cultural conflict. It felt much more like a Trade Pact story, and since my enjoyment of those stories is why I was reading this trilogy in the first place, I'm totally fine with that. Unfortunately, it's also infuriating because Czerneda forecloses on any answers to the historical questions I've been puzzling over for the whole trilogy. The details of how are spoilers, so I won't get into it except to say that I thought it was a rather cheap and unsatisfying device.

If you came this far in this trilogy, you may as well read the ending. Some pieces of it were moving, and I did enjoy the last part of the story. But this whole trilogy focused on the parts of Aryl's life that I found the least interesting and consistently avoided the questions that Marcus was asking and that I really wanted answered. I hope the following Reunification trilogy revisits this world and provides some resolution.

This is the last book of its trilogy, but there is a sequel trilogy to the earlier Trade Pact trilogy in the same universe that starts with This Gulf of Time and Stars, and it promises to tie the two trilogies together.

Rating: 5 out of 10

Andrew Cater: Seems like I only ever write something at CD release time ... Buster release 2 is happening round about now

17 November, 2019 - 06:28
Updating CD images again for Buster release 2. I've spent a happy day chasing down various CD images and discovering my ignorance once again :)

Thanks to Sledge for having me around and also to Schweer - who wasn't feeling so well. No RattusRattus or Isy this time round - that meant I got to do a little more but I do miss their company.

All looks set fair for CD, DVD images and so on: the Jigdo16GB and the BD-1 isos built from jigdo files also worked well for me. This is getting to be a slick process and it's good to be a part of it.

And another one in a few months, I suspect :)

Molly de Blanc: Rebellion

17 November, 2019 - 02:05

We spend a lot of time focusing on the epic side of free software and user freedom: joys come from providing encrypted communication options to journalists and political dissidents; losses are when IoT devices are used to victimize and abuse.

I think a lot about the little ways technology interacts with our lives, the threats to or successes for user freedom we encounter in regular situations that anyone can find themselves able to understand: sexting with a secure app, sharing  DRM-free piece of media, or having your communications listened to by a “home assistant.”

When I was writing a talk about ethics and IoT, I was looking for these small examples of the threats posed by smart doorbells. False arrests and racial profiling, deals with law enforcement to monitor neighborhoods, the digital panopticon — these are big deals. I remembered something I read about kids giving their neighbor a pair of slippers for Christmas. This sort of anonymous gift giving becomes impossible when your front door is constantly being monitored. People laughed when I shared this idea with them — that we’re really losing something by giving up the opportunity to anonymously leave presents.

We are also giving up what my roommate calls “benign acts of rebellion.” From one perspective, making it harder for teenagers to sneak out at night is a good thing. Keeping better tabs on your kids and where they are is a safety issue. Being able to monitor what they do on their computer can prevent descent into objectively bad communities and behavior patterns, but it can also prevent someone from participating in the cultural coming of age narratives that help define who we are as a society and give us points of connection across generations.

People sneak out. People go places their parents don’t want them to. People stay up late at night reading or playing video games. People explore their sexuality by looking at porn when they’re underage. People do things their parents don’t want them to, and these are things their parents are increasingly able to prevent them from doing using technology.

I met someone at a conference who was talking about potentially installing a camera into the bedroom of their pubescent child — the same kind designed to allow parents to monitor their babies at night — because their child was playing video games when they “should be sleeping.”

This appalled me, but one of the things that really struck me was how casually they said it. Technology made it not a big deal. They already had one in their baby’s room, putting another in seemed simple.

I would happily argue all the epic points that come out of this: creating a surveillance state, normalizing the reality of being monitored, controlling behavior and creating a docile population. These are real threats, but also, seriously, poor sleep hygiene is just a thing teenagers do and it’s okay.

These benign acts of rebellion — staying up later than we’re told to, chatting with our friends when we’re not “supposed to” — are not just important points of cultural connection, but also just important for our own individual development. Making mistakes, doing dumb things, acting the fool, and learning from all of this is important in the process of defining ourselves. Technology should not be used to hinder our personal growth, especially when it offers to many opportunities for us to better explore who we are, or makes it safer for us to continue to rebel in the myriad ways we always have. Rebellion is important to our narratives — it’s certainly integral to mine. I hope that people younger than me don’t lose that because of the fear others hold.

Dirk Eddelbuettel: RcppEigen 0.3.3.7.0

16 November, 2019 - 22:41

A new minor release 0.3.3.7.0 of RcppEigen arrived on CRAN today (and just went to Debian too) bringing support for Eigen 3.3.7 to R.

This release comes almost a year after the previous minor release 0.3.3.5.0. Besides the upgrade to the new upstream version, it brings a few accumulated polishes to the some helper and setup functions, and switches to the very nice tinytest package for unit tests; see below for the full list. As before, we carry a few required changes to Eigen in a diff. And as we said before at the previous two releases:

One additional and recent change was the accomodation of a recent CRAN Policy change to not allow gcc or clang to mess with diagnostic messages. A word of caution: this may make your compilation of packages using RcppEigen very noisy so consider adding -Wno-ignored-attributes to the compiler flags added in your ~/.R/Makevars.

The complete NEWS file entry follows.

Changes in RcppEigen version 0.3.3.7.0 (2019-11-16)
  • Fixed skeleton package creation listing RcppEigen under Imports (James Balamuta in #68 addressing #16).

  • Small RNG use update to first example in skeleton package used by package creation helper (Dirk addressing #69).

  • Update vignette example to use RcppEigen:::eigen_version() (Dirk addressing #71).

  • Correct one RcppEigen.package.skeleton() corner case (Dirk in #77 fixing #75).

  • Correct one usage case with pkgKitten (Dirk in #78).

  • The package now uses tinytest for unit tests (Dirk in #81).

  • Upgraded to Eigen 3.3.7 (Dirk in #82 fixing #80).

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Molly de Blanc: Free software activities, October 2019

16 November, 2019 - 07:32

A belated hello! I was traveling at the end of October and missed this. Apologies!

In October, work was quite busy, though a lot of it was behind-the-scenes stuff I cannot yet update you on. It was also very busy with a very exciting trip I took that had absolutely nothing to do with free software. If you’re ever going to Kyoto or Tokyo and looking for some recommendations for coffee, cocktail bars, restaurants, or general things to do, hmu.

Free software activities (personal)
  • I have regular meetings with Patrick Masson, the general manager of the OSI. We made most of them in October.
  • I did some writing for the OSI. Not all of it is published at this point.
  • I worked on crafting drafts of organizational policies for the OSI, including staffing, travel, and a whistle blower policy. I hope to be able to arrange for an HR specialist or employment lawyer to review these.
  • The OSI has two new board members! In order to make this happen, I contacted all of the nominees for whom I had contact information. I spoke with them about the OSI, the Board and it’s activities, and how they saw their potential involvement. Basically I interviewed a bunch of ~fancy~ people. It was so much fun talking with every one of them and I learned so much during the process.
  • The Debian Community Team had some meetings, wrote some emails, and discussed The Future together and with the greater Debian community.
  • I attended All Things Open and spoke about ethics and IoT devices. My slides were puppy themed.
  • I did some philosophy based writing. I got a  lot out of this and hope you did too.
  • I also found out that my brother’s company does some open source work!
Free software activities (professional)
  • I attended All Things Open and had one of the most awesome tabling experiences I have had to date! It was such a great crowd at ATO! They took all of our stickers!
  • I had a lot of meetings with some more great people. Thank you everyone who made some time for me!
  • We launched a Patent Troll Defense Fund! I cannot thank the donors enough! It’s so inspiring for me to see the community come together to support a project I really believe in.
  • We’ve been doing a lot of work on this Rothschild Imaging thing.
  • We did some fundraising for Linux Application Summit (which happened this week!).
  • I submitted to the Open Source Festival 2020 CfP. And you can too!

Romain Perier: Capitole du Libre 2019

16 November, 2019 - 03:19
Hi !

The Capitole Du Libre is a french event that takes place at INP-ENSEEIHT in Toulouse. It is an event dedicated to free and open source softwares. The Capitole Du Libre 2019 will happen this weekend on 16th-17th November.

There will be a Debian booth with the DebianFrance team, I will be there to help them. A lot of interesting talks will be presented, see the schedule here.

Viveris, my company, is a gold sponsor this year. During the year, some of my contributions are sponsorised by my company. Viveris also sponsorised the demo I am going to present at the Debian booth, many thanks to her !

Raphaël Hertzog: Freexian’s report about Debian Long Term Support, October 2019

15 November, 2019 - 21:26

Like each month, here comes a report about
the work of paid contributors
to Debian LTS.

Individual reports

In October, 214.50 work hours have been dispatched among 15 paid contributors. Their reports are available:

  • Abhijith PA did 8.0h (out of 14h assigned) and gave the remaining 6h back to the pool.
  • Adrian Bunk didn’t get any hours assigned as he had been carrying 26h from September, of which he gave 8h back, so thus carrying over 18h to November.
  • Ben Hutchings did 22.25h (out of 22.75h assigned), thus carrying over 0.5h to November.
  • Brian May did 10h (out of 10h assigned).
  • Chris Lamb did 18h (out of 18h assigned).
  • Emilio Pozuelo Monfort did 46.25h (out of 21.75h assigned at the beginning of the month and 24.5h assigned at the end of the month).
  • Hugo Lefeuvre did 46.5h (out of 22.75h assigned and 23.75h from September).
  • Jonas Meurer didn’t get any hours assigned and gave back the 14.5h he was carrying from September as he did nothing.
  • Markus Koschany did 22.75h (out of 22.75h assigned).
  • Mike Gabriel did 11.75h (out of 10h assigned and 1.75h from September).
  • Ola Lundqvist did 8.5h (out of 8h assigned and 14h from September), thus carrying over 13.5h to November.
  • Roberto C. Sánchez did 8h (out of 8h assigned).
  • Sylvain Beucler did 22.75h (out of 22.75h assigned).
  • Thorsten Alteholz did 22.75h (out of 22.75h assigned).
  • Utkarsh Gupta did 10.0h (out of 10h assigned).
Evolution of the situation

In October Emilio spent many hours bringing firefox-esr 68 to jessie and stretch, thus expanding the impact from Debian LTS to stable security support. For jessie firefox-esr needed these packages to be backported: llvm-toolchain, gcc-mozilla, cmake-mozilla, nasm-mozilla, nodejs-mozilla, cargo, rustc and rust-cbindgen.
October was also the month were we saw the first paid contributions from Utkarsh Gupta, who was a trainee in September.

Starting in November we also have a new trainee, Dylan Aïssi. Welcome to the team, Dylan!

We currently have 59 LTS sponsors sponsoring 212h per month. Still, as always we are welcoming new LTS sponsors!

The security tracker currently lists 35 packages with a known CVE and the dla-needed.txt file has 35 packages needing an update.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

Keith Packard: picolibc-1.1

15 November, 2019 - 13:39
Picolibc Version 1.1

Picolibc development is settling down at last. With the addition of a simple 'hello world' demo app, it seems like a good time to stamp the current code as 'version 1.1'.

Changes since Version 1.0
  • Semihosting helper library. Semihosting lets an application running under a debugger or emulator communicate through the debugger or emulator with the environment hosting those. It's great for platform bringup before you've got clocking and a serial driver. I'm hoping it will also make running tests under qemu possible. The code works on ARM and RISC-V systems and offers console I/O and exit() support (under qemu).

  • Hello World example. This is a stand-alone bit of code with a Makefile that demonstrates how to build a complete application for both RISC-V and ARM embedded systems using picolibc after it has been installed. The executables run under QEMU using a provided script. Here's all the source code you need; the rest of the code (including semihosting support) is provided by picolibc:

    #include <stdio.h> #include <stdlib.h>

    int main(void) { printf("hello, world\n"); exit(0); }

  • POSIX file I/O support. For systems which have open/close/read/write, picolibc's tinystdio can now provide stdio functions that use them, including fopen and fdopen.

  • Updated code from newlib. I've merged current upstream newlib into the tree. There were a few useful changes there, including libm stubs for fenv on hosts that don't provide their own.

Where To Get Bits

You can find picolibc on my personal server's git repository:

https://keithp.com/cgit/picolibc.git/

There's also a copy on github:

https://github.com/keith-packard/picolibc

If you like tarballs, I also create those:

https://keithp.com/picolibc/dist/

I've create tags for 1.1 (upstream) and 1.1-1 (debian packaging included) and pushed those to the git repositories.

Filing Issues, Making Contributions

There's a mailing list at keithp.com:

https://keithp.com/mailman/listinfo/picolibc

Or you can file issues using the github tracker.

Kees Cook: security things in Linux v5.3

15 November, 2019 - 08:36

Previously: v5.2.

Linux kernel v5.3 was released! I let this blog post get away from me, but it’s up now! :) Here are some security-related things I found interesting:

heap variable initialization
In the continuing work to remove “uninitialized” variables from the kernel, Alexander Potapenko added new “init_on_alloc” and “init_on_free” boot parameters (with associated Kconfig defaults) to perform zeroing of heap memory either at allocation time (i.e. all kmalloc()s effectively become kzalloc()s), at free time (i.e. all kfree()s effectively become kzfree()s), or both. The performance impact of the former under most workloads appears to be under 1%, if it’s measurable at all. The “init_on_free” option, however, is more costly but adds the benefit of reducing the lifetime of heap contents after they have been freed (which might be useful for some use-after-free attacks or side-channel attacks). Everyone should enable CONFIG_INIT_ON_ALLOC_DEFAULT_ON=1 (or boot with “init_on_alloc=1“), and the more paranoid system builders should add CONFIG_INIT_ON_FREE_DEFAULT_ON=1 (or “init_on_free=1” at boot). As workloads are found that cause performance concerns, tweaks to the initialization coverage can be added.

pidfd_open() added
Christian Brauner has continued his pidfd work by rceating the next needed syscall: pidfd_open(), which takes a pid and returns a pidfd. This is useful for cases where process creation isn’t yet using CLONE_PIDFD, and where /proc may not be mounted.

-Wimplicit-fallthrough enabled globally
Gustavo A.R. Silva landed the last handful of implicit fallthrough fixes left in the kernel, which allows for -Wimplicit-fallthrough to be globally enabled for all kernel builds. This will keep any new instances of this bad code pattern from entering the kernel again. With several hundred implicit fallthroughs identified and fixed, something like 1 in 10 were missing breaks, which is way higher than I was expecting, making this work even more well justified.

x86 CR4 & CR0 pinning
In recent exploits, one of the steps for making the attacker’s life easier is to disable CPU protections like Supervisor Mode Access (and Execute) Prevention (SMAP and SMEP) by finding a way to write to CPU control registers to disable these features. For example, CR4 controls SMAP and SMEP, where disabling those would let an attacker access and execute userspace memory from kernel code again, opening up the attack to much greater flexibility. CR0 controls Write Protect (WP), which when disabled would allow an attacker to write to read-only memory like the kernel code itself. Attacks have been using the kernel’s CR4 and CR0 writing functions to make these changes (since it’s easier to gain that level of execute control), but now the kernel will attempt to “pin” sensitive bits in CR4 and CR0 to avoid them getting disabled. This forces attacks to do more work to enact such register changes going forward. (I’d like to see KVM enforce this too, which would actually protect guest kernels from all attempts to change protected register bits.)

additional kfree() sanity checking
In order to avoid corrupted pointers doing crazy things when they’re freed (as seen in recent exploits), I added additional sanity checks to verify kmem cache membership and to make sure that objects actually belong to the kernel slab heap. As a reminder, everyone should be building with CONFIG_SLAB_FREELIST_HARDENING=1.

KASLR enabled by default on arm64
Just as Kernel Address Space Layout Randomization (KASLR) was enabled by default on x86, now KASLR has been enabled by default on arm64 too. It’s worth noting, though, that in order to benefit from this setting, the bootloader used for such arm64 systems needs to either support the UEFI RNG function or provide entropy via the “/chosen/kaslr-seed” Device Tree property.

hardware security embargo documentation
As there continues to be a long tail of hardware flaws that need to be reported to the Linux kernel community under embargo, a well-defined process has been documented. This will let vendors unfamiliar with how to handle things follow the established best practices for interacting with the Linux kernel community in a way that lets mitigations get developed before embargoes are lifted. The latest (and HTML rendered) version of this process should always be available here.

Those are the things I had on my radar. Please let me know if there are other things I should add! Linux v5.4 is almost here…

© 2019, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.

Uwe Kleine-König: Philips Hue Bridge v2.1

14 November, 2019 - 18:00

I recently bought a Hue Bridge to experiment a bit with Zigbee and 802.15.4. Following two posts for the hardware version 2.0 and some comments about the differences to version 2.1 I was able to get shell access on my 2.1 hardware.

As there is up to now no complete guide I describe here, what I did:

Opening the case is straigth forward. Just remove the two lower nubsis at the bottom and unscrew the two torx screws; then carefully unclip the bottom.

The header on the left side on the image gives access to the UART. Pin 1 is ground, pin 4 is the board's RX (so connect your adapter's TX) and pin 5 is the board's TX. The voltage level is 3.3 V. Note however that the board doesn't boot (after a cold start) with its TX connected so leave that unconnected until after the bootloader is up.

To stop the bootloader short the test point marked with a red cycle on the image near the NAND flash to ground for a few seconds when plugging in the power cord. I used pin 14 of SJ8 (the unassembled header on the left above the factory reset button). This makes the bootloader fail to read the kernel image from NAND and then drop to a shell.

Then after connecting RX you can do there:

setenv bootdelay 3
setenv security '$5$uaS2DwZOsch$2H76tTh0yLzH8LfIkg7z1icoYZLeLGV.Gpo3quhZSe2'
saveenv
reset

After that the machine boots up and you can login with username root and password lala. (If you want a different password, set security in U-Boot accordingly. I did

mkpasswd -m sha-256

to generate the necessary string.)

I already asked Philips for the GPL code running on that machine, didn't hear back from them yet.

I will keep you updated on any progress here.

Norbert Preining: Python 3 deprecation imminent

13 November, 2019 - 15:23

OSS Journal, November 2026. In less than two month, with the end of the year 2026, Python 3 will be deprecated and will not obtain any further security updates. Despite the announcement of deprecation back in summer 2020, shortly after the deprecation of Python 2, still thousands of software projects, in particular in data science, seem to be still based on Python 3.

After the initially quick uptake of Python 4, announced in June 2020, most developers have switched to the clearly superior version which resolves long-standing discrepancies in the Python variable semantics. Unfortunately, Python 4 is not backward compatible with Python 3 (not to think of Python 2).

In a recent interview with the OSS Journal, the Python Head Developer stated:

The future is with Python 4 – we have worked hard to make Python 4 the best programming language out there, and we expect it to serve the community for a long future. Having announced the deprecation of Python 3 well in advance (5 years ago), we expect everyone to have updated their code by now.

The Python developer community has enthusiastically embraced Python 4, and we see no reason to prolongue the outdated Python 3 language just for a few data scientists.

The Python 3 deprecation has created a whole new branch of companies providing only Python upgrade services, but despite the abundance of these services, many programs are still available only for Python 3, some – like Calibre – even only for Python 2.

So let us use the remaining month to fix the billions of lines of code still not compatible with Python 4, for a better future! Rest assured, it will be the last incompatible Python upgrade (for now).

Dirk Eddelbuettel: RcppAnnoy 0.0.14

12 November, 2019 - 19:13

A new minor release of RcppAnnoy is now on CRAN, following the previous 0.0.13 release in September.

RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours—originally developed to drive the famous Spotify music discovery algorithm.

This release once again allows compilation on older compilers. The 0.0.13 release in September brought very efficient 512-bit AVX instruction to accelerate computations. However, this could not be compiled on older machines so we caught up once more with upstream to update to conditional code which will fall back to either 128-bit AVX or no AVX, ensuring buildability “everywhere”.

Detailed changes follow below.

Changes in version 0.0.14 (2019-11-11)
  • RcppAnnoy again synchronized with upstream to ensure builds with older compilers without AVX512 instructions (Dirk #53).

  • The cleanup script only uses /bin/sh.

Courtesy of CRANberries, there is also a diffstat report for this release.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Mary Poppins

11 November, 2019 - 11:04

Review: Mary Poppins, by P.L. Travers

Series: Mary Poppins #1 Illustrator: Mary Shepard Publisher: Houghton Mifflin Harcourt Copyright: 1934 Printing: 2014 ISBN: 0-544-57475-3 Format: Kindle Pages: 202

I read this book as part of Mary Poppins: 80th Anniversary Collection, which includes the first four books of the series.

I have a long-standing irritation with movies that become so famous that they swallow the book on which they were based, largely because of three examples from my childhood: Bambi (the book is so much better), The Wizard of Oz (the book is... not really better, but the rest of the series certainly is), and Mrs. Frisby and the Rats of NIMH (the book is so much more). That irritation is sometimes misplaced, however. Even Disney has been known to make a mediocre book into a better movie on occasion (The Hundred and One Dalmations). When Mary Poppins came up recently (a movie I adored as a kid), I vaguely remembered having read the book long ago, couldn't remember anything about it, and wondered what side of the fence it would come down on. Since an anniversary collection of the first four books was free with Amazon Prime, it was easy to find out.

Answer: perhaps the series improves in later books, but the movie is totally different in tone from the first book, and much better.

I am surprised that I'd forgotten as much about this book as I had, even though it's been at least thirty years since I've read it, since it is extremely odd. I suspect the highly episodic structure is to blame. Mary Poppins never develops into a proper story; instead, it's a series of vignettes about the titular character, the Banks family, and other people on Cherry-Tree Lane. Some of these stories will be familiar from the movie (Uncle Albert floating up into the air because of his laughter). Some will definitely not be, such as a (very brief) trip around the world via a magical compass, a visit from a star who is Christmas shopping, or a truly bizarre birthday celebration for Mary Poppins in the zoo. Unlike the movie, there is no unifying theme of Mary Poppins fixing the Banks's family problems; quite to the contrary, she seems entirely uninterested and even oblivious to them.

This is not Julie Andrews's kind, gentle, and magically competent nurse. There aren't two separate advertisements for her job; this Mary Poppins appears after Mrs. Banks sent letters to the papers advertising for a position and blithely dismisses her request for references. She is neither kind nor gentle, although by the end of the book one gets the feeling she's brought a sort of gruff stability to the household. Like the movie character, she does take the children on adventures, but they seem almost accidental, a side effect of being around Mary Poppins and thus inadvertantly involved in her business (which she rarely, if ever, explains). It's a more intimidating and fae strangeness, only slightly explained by a chapter that reveals that all children know how to talk to the sun and the wind and the animals but forget when they turn one, except for Mary Poppins. (Ode: Intimations of Immortality has a lot of depressing philosophy to answer for.)

Perhaps the oddest difference from the movie for me is that Travers's Mary Poppins is endlessly vain. She's constantly admiring herself in shop windows or finding just the right clothes, much to the frequent boredom of the children. It's an amusing take on a child's view of adult shopping trips, but the vanity and preening feels weirdly out of place for such a magical character.

There is no change in the Banks household in this book; perhaps there is more material in the later books. (The whole series appears to be eight volumes.) When the wind changes, Mary Poppins disappears as mysteriously as she appears, not even saying goodbye, although she does leave some gifts. By that point, Jane and Michael do seem fond of her, although I'm not entirely sure why. Yes, there are adventures, but outside of them, and even during them, Mary Poppins is short, abrupt, demanding, and fond of sharp and dismissive aphorisms. Gregory Maguire proclaims his preference for the books in the foreword on the grounds that they show more glimmers of mystery and danger, and I can see that if I squint. But I mostly found her unpleasant, dictatorial, irritating, and utterly unwilling to explain anything to curious children.

On this point, I'll dare to disagree with Maguire and prefer the Disney version.

A few of the stories here were fun and entertaining in ways not provided by the movie, particularly "Miss Lark's Andrew" (the successful escape of a neighbor dog from enforced isolation and unnatural pampering) and "Christmas Shopping" (I do hope the Pleiades liked their presents!). But when I get the urge for Mary Poppins, I think I'll turn to the movie with no regrets. This is an interesting curiosity, and perhaps subsequent books add more depth (and make Mary less obnoxious), but I don't think it's worth seeking out.

Followed by Mary Poppins Comes Back.

Rating: 6 out of 10

Keith Packard: picolibc-hello-world

11 November, 2019 - 05:54
Picolibc Hello World Example

It's hard to get started building applications for embedded RISC-V and ARM systems. You need to at least:

  1. Find and install the toolchain

  2. Install a C library

  3. Configure the compiler for the right processor

  4. Configure the compiler to select the right headers and libraries

  5. Figure out the memory map for the target device

  6. Configure the linker to place objects in the right addresses

I've added a simple 'hello-world' example to picolibc that shows how to build something that runs under qemu so that people can test the toolchain and C library and see what values will be needed from their hardware design.

The Source Code

Getting text output from the application is a huge step in embedded system development. This example uses the “semihosting” support built-in to picolibc to simplify that process. It also explicitly calls exit so that qemu will stop when the demo has finished.

#include <stdio.h>
#include <stdlib.h>

int
main(void)
{
    printf("hello, world\n");
    exit(0);
}
The Command Line

The hello-world documentation takes the user through the steps of building the compiler command line, first using the picolibc.specs file to specify header and library paths:

gcc --specs=picolibc.specs

Next adding the semihosting library with the --semihost option (this is an option defined in picolibc.specs which places -lsemihost after -lc):

gcc --specs=picolibc.specs --semihost

Now we specify the target processor (switching to the target compiler here as these options are target-specific):

riscv64-unknown-elf-gcc --specs=picolibc.specs --semihost -march=rv32imac -mabi=ilp32

or

arm-none-eabi-gcc --specs=picolibc.specs --semihost -mcpu=cortex-m3

The next step specifies the memory layout for our emulated hardware, either the 'spike' emulation for RISC-V:

riscv64-unknown-elf-gcc --specs=picolibc.specs --semihost -march=rv32imac -mabi=ilp32 -Thello-world-riscv.ld

with hello-world-riscv.ld containing:

__flash = 0x80000000;
__flash_size = 0x00080000;
__ram = 0x80080000;
__ram_size = 0x40000;
__stack_size = 1k;
INCLUDE picolibc.ld

or the mps2-an385 for ARM:

arm-none-eabi-gcc --specs=picolibc.specs --semihost -mcpu=cortex-m3 -Thello-world-arm.ld

with hello-world-arm.ld containing:

__flash =      0x00000000;
__flash_size = 0x00004000;
__ram =        0x20000000;
__ram_size   = 0x00010000;
__stack_size = 1k;
INCLUDE picolibc.ld

Finally, we add the source file name and target elf output:

riscv64-unknown-elf-gcc --specs=picolibc.specs --semihost
-march=rv32imac -mabi=ilp32 -Thello-world-riscv.ld -o
hello-world-riscv.elf hello-world.c

arm-none-eabi-gcc --specs=picolibc.specs --semihost
-mcpu=cortex-m3 -Thello-world-arm.ld -o hello-world-arm.elf
hello-world.c
Summary

Picolibc tries to make things a bit simpler by offering built-in compiler and linker scripts along with default startup code to try and make building your first embedded application easier.

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้