Rescue System » History » Version 62

« Previous - Version 62/202 (diff) - Next » - Current version
Samuele Di Lernia, 06/30/2016 04:43 PM


Rescue System

A minimal Linux system, that is loaded before the main system.

- Select a kernel in /boot on the main rootfs based on priority, verify checksums
- Enable re-flashing the rootfs from a given URL

Install

rescue system should be installed by default on devel/testing feeds, which can be checked by:

$ sed  -n 's/.*\(rescue-.*\).*/\1/p' /proc/cmdline

install it explicitly

opkg install rescue-install

Reflash of Rootfs

WARNING

You can easily destroy your device following these step. This has not yet been tested widely and is intended to be done in a laboratory environment where you have access to the boot loader and can restore the device by other means. Make sure you backup all your settings, metering data, etc. In case of problems request help on the irc channel, because this will definitely VOID YOUR WARRANTY

# opkg install rescue-utils
# reflash_rootfs -h
Usage:
  reflash_rootfs [-u url] [-t feed]
Options:
  -t use a predefined url based on devel/testing/production
  -l list available feeds
  -u specify url to desired rootfs explicitly
  -h print this help

Will create a reflash request file. This file serves as a command to the
rescue system, to erase the existing rootfs and re-install it from given url.

This will create a file /var/lib/rescue-system/reflash_request.conf. That folder serves as mailbox for communication with the rescue-system

ROOTFS_IMAGE_URL=url
ROOTFS_IMAGE_FILE=path   (use either IMAGE_URL or IMAGE_FILE)
ROOTFS_IMAGE_BYTES=number (optional)
ROOTFS_SIGNATURE_FILE=path (optional)
ROOTFS_CHECKSUM=number (optional)

Selecting target kernel based on priority

Inspired by the Freedesktop Bootloader Specification
http://freedesktop.org/wiki/Specifications/BootLoaderSpec/

Idea is to drop/remove files into a folder without any re-configuration of the boot loader

The kernel to be booted is selected based on kernel version. Higher version means higher priority. The boot entry has to follow a certain naming schema. The description is ignored.

BOOT_ENTRY_NAME := 'linux-' VERSION '-description' '.conf'
VERSION := x.x.x | x.x

See https://git.digitalstrom.org/bsp/rescue-system/blob/master/src/dss-boot#L251 for how the selection mechanism works

Boot entry files support the following keywords:

IMAGE_FILE[exclusive] := path to zImage/uImage relative to /boot folder
IMAGE_VOLUME_DEVICE[exclusive] := /dev/mtd1, /dev/mmmcblk0p1 if kernel is stored in a raw partition
IMAGE_FORMAT[optional] := 'zImage' / 'uImage' if not obvious from image file name
IMAGE_CHECKSUM[optional] := sha256sum of the image
DTB_FILE := path to devicetree file, relative to /boot folder
DTB_CHECKSUM[optional] := sha256sum of device tree file
ARGUMENTS[optional] := use this a cmdline instead of forwarding u-boot cmdline
ATAGS[optional] := kernel uses legacy ATAGS mechanism instead of device tree files

Manually install your custom built kernel

Copy zImage and devicetree file into /boot folder of the main root filesystem. Then create a boot-entry in folder /boot/entries

$ cat linux-4.4.1-custom.conf
IMAGE_FILE=zImage-4.4.1
DTB_FILE=devicetree-zImage-imx6q-dss11e6x.dtb

Manually disable a kernel to be selected

If the boot entry file does not match the 'linux-version-description.conf' pattern it will be ignored.

$ cd /boot/entries
$ mc linux-4.1.19.conf #linux-4.1.19.conf

Testing

The rescue system itself needs to be updated. To always have a bootable system, the production system must be bootable without rescue-system. That means the u-boot must have a default kernel to select which is able to mount the rootfs.
This is a precaution against power failures during upgrade. This non-rescue boot path is termed "factory or fallback kernel boot"

Non rescue-system boot works

To test this, we need to disable rescue. Then either observe the console or grep /proc/cmdline as above Install

# opkg remove --force-depends rescue-install

Temporarily disable the rescue system

We have two sorts of boot loaders, those with and those without filesystem support. The older models (dss11/d11-1gb) are those without filesystem support.

With filesystem support, the boot loader will check the rescue partition for a boot.scr file, if we rename that file rescue is disabled:

# mount /dev/mmcblk0p3 /mnt/rescue
# (cd /mnt/rescue; mv boot.scr #boot.scr)

Without filesystem support we have to modify the u-boot environment. There is a special variable to enable rescue

# fw_setenv bootsel rescue  (enable)
# fw_setenv bootsel         (disable)

That's all folks

Permanently remove rescue-system

dss11-sdc

root@SS-sdc: # fw_setenv bootsel   # <- removes rescue from the boot order, u-boot will not try to boot rescue
root@SS-sdc: # cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00020000 00020000 "bootstrap" 
mtd1: 00040000 00020000 "uboot" 
mtd2: 00040000 00020000 "uboot-env" 
mtd3: 00200000 00020000 "kernel-rescue" 
mtd4: 00200000 00020000 "kernel-prod" 
mtd5: 02000000 00020000 "rootfs-rescue" 
mtd6: 0c800000 00020000 "rootfs-prod" 
mtd7: 01360000 00020000 "config" 
root@dSS-sdc:~# flash_erase /dev/mtd5 0 0
Erasing 128 Kibyte @ 1fe0000 -- 100 % complete 
root@dSS-sdc:~# flash_erase /dev/mtd3 0 0
Erasing 128 Kibyte @ 1e0000 -- 100 % complete

analogous for dss11-1gb

root@dSS-TT-UpdateTester-dss11-1gb:/home/dssadmin# fw_setenv bootsel
root@dSS-TT-UpdateTester-dss11-1gb:/home/dssadmin# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00400000 00080000 "kernel-rescue" 
mtd1: 00400000 00080000 "kernel-prod" 
mtd2: 02000000 00080000 "rootfs-rescue" 
mtd3: 3d000000 00080000 "rootfs-prod" 
mtd4: 00800000 00080000 "config" 
mtd5: 00080000 00010000 "spi32766.0" 
mtd6: 3b4f0000 0007e000 "dss11-1gb-rootfs" 
root@dSS-TT-UpdateTester-dss11-1gb:/home/dssadmin# flash_erase /dev/mtd2 0 0
Erasing 512 Kibyte @ 1f80000 -- 100 % complete
root@dSS-TT-UpdateTester-dss11-1gb:/home/dssadmin# flash_erase /dev/mtd0 0 0
Erasing 512 Kibyte @ 380000 -- 100 % complete

Default environment

Modifying the u-boot environment means erasing, then rewriting it. In the better case (dss11-sdc) we have two environments and we always write the new before deleting the old one. But in case of dss-1gb we have to erase before writing the new one. This means, due to a power failure we might lose our environment.

fw_setenv is supposed to have the same default environment compiled in, as the u-boot does.... tbd.
open points:
- default u-boot env selects the factory kernel by default, mechanism to re-install rescue if not selected
- mac address stored in u-boot environment, #12048

env default -a

http://stackoverflow.com/questions/20621937/how-do-i-clear-environment-variables-previously-saved-with-u-boot

Default rootfs links

A folder on the production rootfs serves as mailbox to communicate with the rescue-system. If the production rootfs is corrupted, rescue download a new image from an URL it has compiled in. These links should be tested

Delete the production rootfs from rescue, then reboot and see what happens

dss11-1gb/dss11-sdc:

flash_erase /dev/mtd3 0 0

dss11e/dss20:

dd if=/dev/null of=/dev/mmcblk0p5 bs=512 count=1024

Installation of rescue-system itself on on dss11-e

Rescue system is not installed in the factory, at least for dss11-1gb/dss11-sdc/dss11-e. So we must test the initial installation during an upgrade. The critical part is formatting the rescue-partition on dss11-e.

The idea is to downgrade to version before rescue-system is installed, wipe out the rescue partition, then reboot and upgrade:

untested

# reflash_rootfs -u http://update.aizo.com/feeds/digitalstrom-production-eglibc/images/dss11e/digitalstrom-production-rootfs-1.10.4-dss11e.ext3
# reboot

We still need rescue-system at that point to do the upgrade, after downgrade is complete, destroy the filesystem where rescue-system is installed

# dd if=/dev/zero of=/dev/mmcblk0p3 count=10 bs=512
# reboot (optional)

Then do an upgrade to current production and follow the update messages. Also reboot, to verify that rescue-system is booting correctly.

Links

Design/Dump of ideas

http://ccweb.cc/edit/p/rescue_system
https://tools.digitalstrom.org/pad/p/rescue

Repository:
https://git.digitalstrom.org/bsp/dss-rescue-tools

Intial commit by era:
https://gitorious.digitalstrom.org/~era/dss-oe/eras-dss-oe/commits/dss-rescue-system

rescue_gdansk_2018.pdf (1.25 MB) Andreas Fenkart, 04/18/2018 10:22 AM