Code Sign & Certificates

The updates shall be cryptographically signed.
The update shall be possible without network (USB update)
Sign kernel boot-entries (optional?)


Instead of documenting indvidual steps there is repository with script to create all needed artifacts.

Related designs

Strong Distribution HOWTO
V. Alex Brennen ()

  • chapter one is worth reading, good introduction to the issues we face


Updates are signed by key stored on SmartCard

When using a tamper proof hardware, the private key can't be stolen by an attacker

Trusted Key(SmartCard)   ---- export public key   ---->   Rescue System
        \                                                     /           
         \ signs                                             /   verifies
          \                                                 /       
            ---------->    firmware update    <------------- 
No network access required during upgrade When signing updates user needs to enter pin
Private key can not be extracted from SmartCard Adding/Removing keys is hard

SmartCard loss might eventually happen, but the card locks itself after 3 failed attempts to enter the pin. Eventually the public key should be removed from the static keyring

An attacker might try to trick a devop into signing an incorrect update, e.g. build server is compromised. If we now the hash of what has been signed, we can blacklist that hash and distribute it with rescue system update. Mind that valid rootfs with exploitable security flaws can be installed eternally as well, so they should be blacklisted too.

Signing requires entering the pin. Only production/fieldtest updates shall use a SmartCard to sign. Daily / (Testing?) builds still use a private key stored locally on the builder.

SmartCards keys can be backed up / cloned by generating private/public keys on the host computer then transferring the private key to several cards. Printouts of the private key can be used as backups.

To rotate keys they need to be added/removed from the keyring in the rescue system. Using a new key requires that the latest rescue system is installed on the target. If the rescue system is outdated, does not contain the new key, upgrades of the rootfs signed with the new key will fail. In this case users must install interim versions that are signed with a key known to the legacy rescue-system and still trustworthy.


OpenSSL might come to mind first, but GnuPG offers the same cryptography algorithms. Feature wise both offer what we need, while GnuPG - IMHO - is far easier to use. Many Linux distributions use it for signing their updates. So we are not doing much wrong in deciding in favor of GnuPG. GnupG 1.x is designed for embedded, it is hard to beat footprint wise, see #Image-footprint. #Image-footprint


Move keys to card:

- taken from wiki page

- key length now up to 4096 bits
- key storage for up to three key pairs

- RSA keys of 1024, 2048, 3072 and 4096 bit are supported.
- store three RSA key pairs. All keys use the same identity but are used for different purposes: authentication, encryption and signing.
- tamper resistant smart card. Nitrokey Start is implemented in the microprocessor.

Size comparison of Nitrokey and Gemalto Reader with GnuPG kernel concepts card inside

kernel concepts card with SIM breakout

Both work out of the box. Nitrokey is about 30% smaller. Nitrokey offers the same features based on OSS as kernel concepts does.

Both designs rely on BasicCard, with an with OpenGPG applet. BasicCard shall also be the basis for the German Health Card. NitroKey Pro embeds a BasicCard and a USB MCU serving as a card reader, in a compact form factor. Mind that the reader is part of security concept, since it has access to the pin. It's probably easier to attack then the card itself.
Discussion NitroKey/KernelConcepts.

kernelconcepts nitrokey pro
(+) Tamper resistant, rebranded BasicCard (+) Tamper reistant, embedded BasicCard inside
(+) card/reader separate, can use reader with keypad (+) about 30% smaller in length, very robust, intended to be added to your physical keyring (real keys)
(+) a tick cheaper cards (35.- Euro incl. reader) (-) more expensive (49 Euro)
(-) plastic cover of gemalto looks a bit fragile (+) hardware bom uses STM32F103
(?) proprietary application offering weblogin via one-time-password and keystore
(-) Need to trust the reader (+) You can inspect the reader code

not evaluated:
NitroKey HSM since code signing is yet limited, interesting though since it support elliptic curve cryptography
- OSS software project, requires to buy platform yourself
- platforms supported (STM32F103, STM32F030, MKL27Z)
- rumors say YubiKey / NitroKey is based on second source project FST-01
- handout to members of the FSF Europe, not intended for sale
- like kernelconcepts probably a javacard with the OpenGPG applet
- uses security chip NXP A700x
- closed source components
- OpenGPG applet for YubicoNEO

Key length

None of the smart card supports Elliptic Curve, the algorithm likely to replace RSA.
- 4096 offers very little security over 2048, while consuming more CPU, memory.
- NIST claims 2048 to be secure till 2030
- most others don't

Evtl. still use 4096 since our design depends on it so heavily
- how much time at91 needs to verify kernel signature

Hash length

sha256 should be enough according most recommendations.

TODO: what is the impace of 384


Create signing/encryption/authenticate keys

gpg2 --full-gen-key

Remote GPG agent

TLS support for update server (evtl. key server)

Signature verification is opt-in. With root access on the platform any FW can be flashed. For example by creating an malicious update instruction Rescue_System

The server hosting the upgrade instructions is part of the security design. If these packages are served by our update server then that server must use https and a valid x509 certificate. Evtl. it is sufficient to sign the packages.gz, but that's not discussed here.

TLS still provides secrecy what patches are installed and what vulnerabilities the client still has while installing the update.

Public key infrastructure(pki) / x.509 certificates

It's easier to generate x509 certificates using OpenSSL. It's probably easier to buy a certificate for the keyserver from a certification provider. But we can also create our own root certificate and install the public key in our firmware images and rescue system. End users will not use those servers directly, since those are the only used by our firmware.

These webservers should have certificates:

  CA -> pki structures: (certificate revocation lists, policy) 
   | -> gnupg key server
   \ -> update server (optional)

The self-signed certificate warning is not visible to users unless the surf explicly to those server, in that case we can advertise our CA root key.
This has the advantage that we can use real short expiry periods on all certificates (e.g. 4 weeks) and rotate them every 2 weeks which limits the amount of damage a stolen certificate can do.

Buying a certificate has the advantage that we don't have to serve the ca structures ourselves.

CRL/OSCP PKI structures / links


Attacks on GnuPG SmartCard signing

Attacker controls builder server, SmartCard signatures

Controlling the builder allows an attacker to inject his own code during regular builds. The final output will be signed as a regular update. Few cues prevent such an attack

  • testing will verify that patches are included in the final output
    It's not enough to replace the rootfs by it's own malicious rootfs. The output still has to pass testing department, open bugs are retested, that means all patches must be present.
  • recipes and builder scripts are under version control
    An attacker needs to the builder recipes after the build is started and revert them before the build is complete, otherwise they risk discovery
  • rootfs must be hosted from the update server, not random server
    If an image is signed it can't be taken back easily. An attacker having a copy of such image, needs to control update server too, since the address of the update server is hard-coded in the target, and they can only be installed through USB otherwise. (*)

Private keys can not be stolen, since it doesn't leave the secure token card. But this doesn't solve the problem what actually gets signed. Good care has to be taken to make the builder server as secure as possible.

  • no password logins
  • building 3rd party software risky (yocto)

(*) If we are aware of such images we can blacklist the hashes of such images in the rescue-system, without revoking the key itself

An attacker can downgrade to a previous version containing a known vulnerability

I a previous FW had security flaws might create an upgrade instruction. Since the FW is signed with a valid key, signature verification will not stop from this. Once the legacy FW is installed the known security issue can be exploited

Hardware attacks

If the card is lost or stolen, different attacks on the card itself are possible.

Image footprint

GnuPG vs. OpenSSL

OpenSSL: openssl (500kB) / (1.5MB)
GnuPG: single binary (800kB)

For TLS verification libssl is sufficient (270kB)

-- double check numbers

GnuPG v2.x vs GnuPG v1.x

Resulting rescue system for dss-1gb

initramfs size
GnuPG v2.x gpg 5911 kBytes
GnuPG v1.x gpg 3464 kBytes
GnuPG v1.x gpgv 2954 kBytes

CPU usage depending on RSA keylength

signature verification on target

  • test in dss11-sdc
  • ten runs / average
  • 86 MByte rootfs file "Angstrom-dss11-devel-image-20160407082717-eglibc-ipk-v2016.04-core-dss11-1gb.rootfs.ubifs"
bits real
1024 42.2s
2048 42.2s
4096 42.2s
  • 1.5 MByte kernel "zImage-4.1.19-r6-at91sam9g20ek-20160525015923.bin"
bits real
1024 0.45s
2048 0.45s
4096 0.46s

signature creation on smartcard

  • enable card to cache pin: gpg2 --edit-card / forcesig
  • needed to enter pin 10 times, max-cache-ttl does not cache pin
  • smartcard connected to x86-64
  • signing kernel image: 1.6MBytes
kernelconcepts nitrokey pro
2048 0.5s 0.6s
4096 2.9s 3.2s

Alternative designs

Private key stored on builder with key rotation

The builder will sign updates using GnuPG key pair without password. The builder key is signed by a trusted key, that is kept secure. That trusted key is best stored on a secure SmartCard which is readily supported by GnuPG. The public key of the trusted entity is exported and included in the rescue system initramfs.

   Trusted Key(SmartCard)    ----  public key builtin ---->   Rescue System
      \               \                                        /      /
       \ signs         \ signed public builder key            /  retrieve current builder key
        \               \                                    /      /
       builder key       ------->    key server   <---------       /
              \                                                   /
               \ signs                                           /  verifies using current builder key
                \                                               /
                 -------->     FW update  & signature    <----- 

In case the builder machine is compromised its key must be revoked, and replaced with a new key. The rescue system will fetch the current public builder key from the key server. This is important since it allows existing installations to verify updates, that are signed with the new un-compromised builder key. Revocation of the builder key is done by uploading a revocation certificate onto the key server.

Installations of updates start with by downloading the current builder key, which can be looked up on the key server by a well known user id, e.g. email address.

builder can sign updates without user interaction requires network connection during update (key server)
key server single point of failure(*)
verification of signed boot entries(kernels) requires network connection

(*) likely update server down as well

GnuPG keyserver support

Probably what most servers use:

Alternative implementation (experimental) is Hockeypuck, written in Go

Good overview


The key server must be trusted. An attacker that gained the builder keys, can use them forever if he suppresses their revocation. This can be achieved by impersonating the key server, e.g. through DNS spoofing. Hence the key server itself must use a valid certificate. Key servers use the Horowitz key protocol (hkp), which is a tiny layer on top of http, hence using TLS shouldn't be an issue.

Use GnuPG to sign tip of your repository

7.4 Git Tools - Signing Your Work

SmartCard readers

- pinpad with support for linux

Links/Stuff need to be sorted:

evtl. find an extension to export smart card remotely:

ohter pkcs#11 smartcard exporting

keycard_big.jpg - keycard_big (56.1 KB) Andreas Fenkart, 06/20/2016 03:23 PM

smartcards.jpg - smartcards (102 KB) Andreas Fenkart, 06/20/2016 03:23 PM