LibreELEC RAID support

Why?

LibreELEC will not support RAID, either hardware or software configurations, as it is considered out of scope for this project. Consider a NAS.

But LibreELEC is Linux-based and it is open, so we could bring RAID support and customize it for our needs.

Requirements

Here are my requirements for RAID and my configuration:

• RAID mirror for two usb hard drives
• email notifications about RAID state
• build everything on OS X
• run S905-based media box with LibreELEC

If you have a different media box, windows you will need to slightly alter installation workflow.

Prerequisites

• working LibreELEC installed on SD card
• Docker for OS X to run Linux toolchain
• knowledge of mdadm

Build LibreELEC

Technically, you could use cross compiler to build LibreELEC on OS X, but I didn’t try it.

I will do that in Debian within Docker container.

Prepare Linux Docker container

I will use Debian as example below.

Create docker and log into it:

docker run -dit --name libreelec-build debian

Log into it:

docker attach libreelec-build

Upgrade repos:

apt-get update && apt-get upgrade

Install build toolchain:

apt-get install gcc make git wget

Install some other build dependencies:

apt-get install bc patchutils bzip2 gawk gperf zip unzip lzop g++ default-jre u-boot-tools texinfo xfonts-utils xsltproc libncurses5-dev xz-utils

To make x86 dtbTool work on x86_64:

dpkg --add-architecture i386
apt-get update
apt-get install libc6-i386 lib32stdc++6

Checkout LibreELEC sources

Firstly you need to find repo with LibreELEC source related to your architecture.

In my case it is (for S905 fork): https://github.com/kszaq/LibreELEC.tv.git

Download repo:

cd ~
git clone https://github.com/kszaq/LibreELEC.tv.git

Checkout specific version:

cd ~/LibreELEC.tv
git tag | less
git checkout X.X.Xy

Build LibreELEC without patches

Build the same version of LibreELEC you have already installed to test if it works well. If it will work well then you could move to next step, if it doesn’t work then you are doing something wrong (wrong platform, architecture, version, repo etc).

Original compilation process is documented here: https://wiki.libreelec.tv/index.php?title=Compile

Download package sources:

PROJECT=S905 ARCH=arm tools/download-tool

Build image:

PROJECT=S905 ARCH=arm make image

Leave it compiling for a night 🙂 Once it will finish you could continue.

Result *.tar.gz image will be in: ~/LibreELEC.tv/target

Copy image to host from Docker container

Find needed image file.

find ~/LibreELEC.tv/target

Run on host (replace {path} with actual *.tar.gz image path) to copy LibreELEC image to ~/Downloads OS X folder:

docker cp libreelec-build:{path} ~/Downloads

Upgrade LibreELEC media and test it

Below is instruction for S905, OS X and for scenario when LibreELEC is installed on SD card.

Unpack new image:

open ~/Downloads/{path}.tar.gz

Insert LibreELEC SD Card. It should be mounted as /Volumes/LIBREELEC.

Mount new LibreELEC image:

hdiutil attach -imagekey diskimage-class=CRawDiskImage {path}.img

Copy all files except dtb.img from mounted image to SD card.

Don’t forget to create /opt folder in SYSTEM image root again if you already have optware-like installed:

cd /Volumes/LIBREELEC
mkdir -p new/opt
mksquashfs new SYSTEM -all-root
rm -rf new

Unmount both LibreELEC SD card and image.

Boot your media box with SD card.

If it works well then you could move to next step.

Patch LibreELEC sources

Fix kernel config

We will need to update this kernel config file (for S905): ~/LibreELEC.tv/projects/S905/linux/linux.arm.conf

Below is the diff file you need to save somewhere, for example, in ~/LibreELEC.tv/projects/S905/linux/linux.arm.conf.patch:

--- .config 2017-05-24 03:03:17.557759250 +0000
+++ .config 2017-05-24 03:17:54.506204792 +0000
@@ -1416,7 +1416,18 @@
 # CONFIG_SCSI_OSD_INITIATOR is not set
 CONFIG_HAVE_PATA_PLATFORM=y
 # CONFIG_ATA is not set
-# CONFIG_MD is not set
+CONFIG_MD=y
+CONFIG_BLK_DEV_MD=y
+# CONFIG_MD_AUTODETECT is not set
+CONFIG_MD_LINEAR=y
+CONFIG_MD_RAID0=y
+CONFIG_MD_RAID1=y
+CONFIG_MD_RAID10=y
+CONFIG_MD_RAID456=y
+# CONFIG_MD_MULTIPATH is not set
+# CONFIG_MD_FAULTY is not set
+# CONFIG_BCACHE is not set
+# CONFIG_BLK_DEV_DM is not set
 # CONFIG_TARGET_CORE is not set
 CONFIG_NETDEVICES=y
 CONFIG_MII=y
@@ -3317,6 +3328,7 @@
 # CONFIG_INTERVAL_TREE_TEST is not set
 # CONFIG_PERCPU_TEST is not set
 # CONFIG_ATOMIC64_SELFTEST is not set
+# CONFIG_ASYNC_RAID6_TEST is not set
 # CONFIG_TEST_STRING_HELPERS is not set
 # CONFIG_TEST_KSTRTOX is not set
 # CONFIG_DMA_API_DEBUG is not set
@@ -3342,7 +3354,12 @@
 CONFIG_SECURITYFS=y
 CONFIG_DEFAULT_SECURITY_DAC=y
 CONFIG_DEFAULT_SECURITY=""
-CONFIG_XOR_BLOCKS=m
+CONFIG_XOR_BLOCKS=y
+CONFIG_ASYNC_CORE=y
+CONFIG_ASYNC_MEMCPY=y
+CONFIG_ASYNC_XOR=y
+CONFIG_ASYNC_PQ=y
+CONFIG_ASYNC_RAID6_RECOV=y
 CONFIG_CRYPTO=y

 #
@@ -3459,7 +3476,7 @@
 #
 # Library routines
 #
-CONFIG_RAID6_PQ=m
+CONFIG_RAID6_PQ=y
 CONFIG_BITREVERSE=y
 CONFIG_RATIONAL=y
 CONFIG_GENERIC_STRNCPY_FROM_USER=y

Apply patch:

patch ~/LibreELEC.tv/projects/S905/linux/linux.arm.conf ~/LibreELEC.tv/projects/S905/linux/linux.arm.conf.patch

Add mdadm package

mdadm building from source code is well described in Linux From Scratch documentation: http://www.linuxfromscratch.org/blfs/view/cvs/postlfs/mdadm.html

mkdir -p ~/LibreELEC.tv/packages/sysutils/mdadm
nano ~/LibreELEC.tv/packages/sysutils/mdadm/package.mk

Put this content into ~/LibreELEC.tv/packages/sysutils/mdadm/package.mk:

PKG_NAME="mdadm"
PKG_VERSION="3.4"
PKG_ARCH="any"
PKG_LICENSE="LGPLv2"
PKG_SITE="http://neil.brown.name/blog/mdadm"
PKG_URL="http://www.kernel.org/pub/linux/utils/raid/$PKG_NAME/$PKG_NAME-$PKG_VERSION.tar.gz"
PKG_DEPENDS_TARGET="toolchain systemd"
PKG_SECTION="system"
PKG_SHORTDESC="mdadm: manager for raid arrays"
PKG_LONGDESC="mdadm is a tool for managing Linux Software RAID arrays."

PKG_IS_ADDON="no"
PKG_AUTORECONF="no"

STRIP="--strip --strip-program=$STRIP"

PKG_MAKE_OPTS_TARGET="SYSCONFDIR=/storage/.config CC=$CC CWFLAGS="
PKG_MAKEINSTALL_OPTS_TARGET="SYSTEMD_DIR=/usr/lib/systemd/system BINDIR=/usr/sbin install-systemd"

Make busybox package depend on mdadm package:

nano ~/LibreELEC.tv/packages/sysutils/busybox/package.mk

Add mdadm in the end of PKG_DEPENDS_TARGET variable:

PKG_DEPENDS_TARGET="... mdadm"

Rebuild LibreELEC

LibreELEC build system will be able to rebuild only your changes without rebuilding everything.

cd ~/LibreELEC.tv

Clear kernel build (we upgraded kernel configuration, so we need to rebuild it):

rm -rf build.*/linux-*

PROJECT=S905 ARCH=arm make image

Result *.tar.gz image will be in: ~/LibreELEC.tv/target

Upgrade LibreELEC media

Upgrade LibreELEC media again in the same way as you already did that but in this case we will have LibreELEC with RAID-capable Linux kernel and mdadm installed.

Configure RAID

Build raid array with mdadm, create partitions, format partitions and mount them via system.d. This out of scope for this article.

System.d unit could be like below.

nano /storage/.config/system.d/storage-mirror.mount

For example:

[Unit]
Requires=local-fs.target
After=local-fs.target

[Mount]
What=/dev/disk/by-uuid/2adbab97-d22e-4fbf-bb74-40520155c52e
Where=/storage/mirror
Type=ext4
Options=noatime

[Install]
WantedBy=multi-user.target

You could find partition UUID with:

blkid

You need to also enable service and start service:

systemctl enable storage-mirror.mount
systemctl start storage-mirror.mount

Note: unit name should be equal to mount point: /storage/mirror -> storage-mirror.mount

Setup email notifications

Install sendmail-like MTA

Install optware-like opkg package manager. Follow this article: https://www.artembutusov.com/libreelec-entware-ng-installation/

Install and configure sendmail. Follow this article: http://www.artembutusov.com/libreelec-sendmail/

Configure mdadm

Here we assume that you have mdadm installed and patched to look for mdadm.conf file in /storage/.config/mdadm.conf.

Mdadm monitor by default will search for sendmail in /usr/sbin/sendmail and will not be able to find sendmail installed as /opt/sbin/sendmail, so MAILADDR, MAILFROM will not work by default.

But PROGRAM will still work and we could create shell wrapper which will do wherever we want.

Edit mdadm config:

nano /storage/.config/mdadm.conf

Put this content:

PROGRAM /storage/.config/mdadm-program.sh

Edit mdadm event handler:

nano /storage/.config/mdadm-program.sh

Put this content (change to your email):

#!/bin/sh

EVENT=$1
DEV=$2
COMPDEV=$3

PROGRAM=mdadm
HOST=$(hostname)
SENDMAIL=/opt/sbin/sendmail
EMAIL=your.email@domain.com

SUBJECT="$PROGRAM@$HOST: $EVENT $DEV $COMPDEV"

BODY="$(cat /proc/mdstat)"

echo -e "Subject: $SUBJECT\n\n$BODY" | $SENDMAIL -t $EMAIL

Make handler executable:

chmod +x /storage/.config/mdadm-program.sh

Each time you will restart media box you will get a new message from mdadm that RAID array was successfully assembled.

If it is not what you want you could slightly fix program (wrap in if condition):

if [ $EVENT != "NewArray" ]; then
    echo -e "Subject: $SUBJECT\n\n$BODY" | $SENDMAIL -t $EMAIL
fi

Test mdadm notifications

Run mdadm --monitor --scan --test to get test message from mdadm monitor.

You will need to temporarily stop mdmonitor service if it is running to to do that:

systemctl stop mdmonitor
mdadm --monitor --scan --test
# press CTRL+C once you get email
systemctl start mdmonitor

But anyway you should be able to get message from mdadm monitor even if your just restart mdadm monitor (email about NewArray event).

Conclusion

You should get a stable RAID support in LibreELEC without any impact on media capabilities.

Unfortunately most LibreELEC media boxes with S905 have slow USB2 so your RAID will not be so fast as it could be on better media box but it is still acceptable if you don’t need super speed. You could share this volume over Samba (Windows), NFS (Linux) or AFP (OS X) and use it over network. You could install software on LibreELEC which will in background download something on the disk.

Please note, RAID5 and RAID6 could be significantly heavier for your media box so I don’t recommend them, but RAID0 (stripe), RAID1 (mirror) and JBOD should be fine.

Please note, most cheap media boxes will have just one or two USB2 host controllers, each USB2 host controller has bandwidth of 450Mbps which will be shared between all connected devices. In my case I got real disk bandwidth around 250Mbps, 125Mbps per each disk in USB docking stating.

If you have media box with USB3 then you should be totally fine.

If somebody is interested I used this cheap USB HDD docking station to connect two hard drives to media box over USB: Inateck FD2002 https://www.amazon.com/gp/product/B00N1KXE9K (this device 100% supports mounting of both drives in the same time). I created RAID mirror on them to be sure that I will not loose any data if one drive will fail. This RAID mirror is shared over network via Samba and AFP and I’m using it also as Time Machine backup disk. See this article: http://www.artembutusov.com/libreelec-as-time-machine-backup-network-device/

I tested it on my media box and it is very stable. Much much more stable than the same disks worked in RAID mirror in NAS Western Digital MyBook DUO. Actually, buggy WD NAS firmware was the main reason for me to run soft RAID on LibreELEC and use LibreELEC as NAS 🙂

The only possible disadvantage of your custom LibreELEC build is that you will have rebuild image manually if you need to update LibreELEC’s version.