By / 05 August 2015 / operations / est. 11 min read

SSD caching with bcache for Linux

This post is older than a year. Consider some information might not be accurate anymore.

This article describes SSD caching for the HDD drive on a new Ubuntu Linux installation. The contents might as well apply to Debian Jessie.

Preconditions

Device: Thinkpad Edge S430, 500 GB hdd, 16 GB ssd, 16 GB ram
OS: Ubuntu 14.04.02 LTS (Desktop)
bcache as ssd cache

Ubuntu base installation

During the installation I partition my hdd drive like this

hdd disk partitioning

root@edge:~# fdisk -l
Disk /dev/sda: 500.1 GB, 500107862016 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux       (1)
/dev/sda2          501758   976771071   488134657    5  Extended
/dev/sda5       960774144   976771071     7998464   83  Linux       (2)
/dev/sda6          501760   960774143   480136192   83  Linux       (3)
Disk /dev/sdb: 16.0 GB, 16013942784 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    31277231    15637592   83  Linux       (4)

/dev/sda1 ⇒ 256 MB boot partition, will not be cached, as precaution for dist-upgrades
/dev/sda5 ⇒ 8 GB for / (sufficient for basic installation and will be later used as swap partition)
/dev/sda6 ⇒ the remaining hdd disk space for migration to bcache, don’t assign a mount point
/dev/sdb1 ⇒ the 16 GB sdd cache

Ubuntu will complain that there is no swap partition and /dev/sda6 has no mount point. These warnings can be ignored from the initial base installation. After you have finished the base installation, you are ready to install bcache.

bcache installation

If you have kernel version 3.10 or greater you should have bcache, eg. Ubuntu distributions since version 14.04. You can lookup the kernel version with

tan@edge:~$ uname -a
Linux edge 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

bcache-tools

The first thing you must do is install bcache-tools. This is the tool that will create and register the block devices to work with bcache. To install bcache-tools you must first add the repository. This is done with the following commands:

sudo add-apt-repository ppa:g2p/storage
sudo apt-get update
sudo apt-get install bcache-tools

sudo apt-get install git
sudo apt-get install build-essential
sudo apt-get install uuid-dev
git clone http://evilpiepirate.org/git/bcache-tools.git
cd bcache-tools
make
sudo make install

If you are using higher ubuntu version like 14.10, bcache-tools are already in the repository.

# install bcache-tools from repo
apt-get install bcache-tools

Concept

bcache has two requirements: # a ‘backing’ device, the device that shall be cached, the empty hard disk or an empty partition on a hard disk # a ‘caching’ device, the device that is used as cache, the SSD, e.g. 16-32 GB contained in hybrid drives

bcache

It is designed around the unique characteristics of flash-based SSDs and uses a hybrid btree/log to track cached extents. It is designed to avoid random writes at all costs. bcache fills up an erase block sequentially and then issues a discard before reusing it. Both write-through and write-back caching are supported. Write-back defaults to off, but can be switched on and off arbitrarily at runtime. The virtual bcache device is created by attaching the backing device and caching device.

Setup

the backing device is /dev/sda6, formatted as ext4
the caching device is /dev/sdb1, formatted as ext4

Cleanup devices

If the devices have already been used for bcache a cleanup is necessary. You may skip this section, if the devices are not used with bcache before.

We put zeroes on the first 4kB of the disk:

# wipe backing device
sudo dd if=/dev/zero if=/dev/sda6 bs=512 count=8
wipefs -a /dev/sda6
# wipe caching device
sudo dd if=/dev/zero if=/dev/sdb1 bs=512 count=8
wipefs -a /dev/sdb1

Create and register the devices

The virtual bcache device is created by attaching the backing device and caching device.

First set up the backing device (the empty partition on your hard drive).

root@edge:/home/tan# make-bcache -B /dev/sda6
UUID:			4bd28ddd-aae2-4b91-8ba9-a7347d365987
Set UUID:		fc051fbd-37b1-4697-9ac9-fdd3e00ed92f
version:		1
block_size:		1
data_offset:	16

Then do the same thing for your caching partition.

root@edge:/home/tan# make-bcache -C /dev/sdb1

If you get an error that there are already non-bcache superblocks on the device(s), you have to remove those errors with the wipefs command (see above cleanup section).

Once the backing device is registered, it will show up in /dev in the form of /dev/bcacheX (where X is a number – such as /dev/bcache0).

You might need to format the devices

mkfs.ext4 /dev/sda6
mkfs.ext4 /dev/sdb1

Show bcache superblock

bcache-super-show prints the bcache superblock of a cache device or a backing device.

root@edge:/home/tan# bcache-super-show /dev/sdb1
sb.magic        ok
sb.first_sector     8 [match]
sb.csum         39497E70770FF75D [match]
sb.version      3 [cache device]
dev.label       (empty)
dev.uuid        dc57ecd7-cc8d-468e-9853-9b0f6214faba
dev.sectors_per_block   1
dev.sectors_per_bucket  1024
dev.cache.first_sector  1024
dev.cache.cache_sectors 31273984
dev.cache.total_sectors 31275008
dev.cache.ordered   yes
dev.cache.discard   no
dev.cache.pos       0
dev.cache.replacement   0 [lru]
cset.uuid       9b1e7bf9-a97a-4785-bacb-0b17189adc08

Attach devices

The devices are now created and registered. You now have to attach the caching and backing devices to enable the caching feature. Here you will need the UUID (a long string of characters) found in /sys/fs/bcache/ (enter the command: ls /sys/fs/bcache and you’ll see the UUID). To attach the devices, you simply use the echo command to add the UUID to the attach file in /sys/block/bcache0/bcache/. The command is:

echo UUID > /sys/block/bcache0/bcache/attach

Where UUID is the actual UUID found in /sys/fs/bcache.

root@edge:/mnt# ls /sys/fs/bcache/
9b1e7bf9-a97a-4785-bacb-0b17189adc08  register  register_quiet
root@edge:/mnt# echo 9b1e7bf9-a97a-4785-bacb-0b17189adc08 > /sys/block/bcache0/bcache/attach

Check bcache status

If bcache was properly setup, check with lsblk.

root@edge:~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 465,8G  0 disk
├─sda1        8:1    0   243M  0 part /boot
├─sda2        8:2    0     1K  0 part
├─sda5        8:5    0   7,6G  0 part
└─sda6        8:6    0 457,9G  0 part
  └─bcache0 251:0    0 457,9G  0 disk /  <b>(1)</b>
sdb           8:16   0  14,9G  0 disk
└─sdb1        8:17   0  14,9G  0 part
  └─bcache0 251:0    0 457,9G  0 disk /  <b>(2)</b>
sr0          11:0    1  1024M  0 rom

bcache0 for the backing device
bcache0 for the caching device

Migrate root partition

create mount points

cd /media
mkdir OLD NEW
mount /dev/sda5 OLD
mount /dev/bcache0 NEW

copy contents from old to new partition

rsync -a OLD/ NEW/

grub install

mount /dev/sda1 NEW/boot
mount -o bind /dev NEW/dev
mount -t proc none NEW/proc
mount -t sysfs none NEW/sys
chroot NEW
ls -l /dev/disk/by-uuid/ | grep bcache0     <b>(1)</b>
vim NEW/etc/fstab                           <b>(2)</b>
update-grub                                 <b>(3)</b>
grub-install /dev/sda

find out the UUID of /dev/bcache0
replace the previous UUID of / with the new UUID ⇒ /dev/bcache0
creates a new grub.cfg with changed values from /etc/fstab

Check caching

After the grub installation, the next reboot should start from the backing device with bcache using the sdd as caching device.

To check to see if the caching is working, open up a terminal window and issue the command:

caching stats

tan@edge:~$ tail /sys/block/bcache0/bcache/stats_total/*
==> /sys/block/bcache0/bcache/stats_total/bypassed <==
294M
==> /sys/block/bcache0/bcache/stats_total/cache_bypass_hits <==
3041
==> /sys/block/bcache0/bcache/stats_total/cache_bypass_misses <==
0
==> /sys/block/bcache0/bcache/stats_total/cache_hit_ratio <==
64
==> /sys/block/bcache0/bcache/stats_total/cache_hits <==
7398
==> /sys/block/bcache0/bcache/stats_total/cache_miss_collisions <==
0
==> /sys/block/bcache0/bcache/stats_total/cache_misses <==
4069
==> /sys/block/bcache0/bcache/stats_total/cache_readaheads <==
0

Reuse old root as swap

/dev/sda5 is no longer used and with 8 GB fits perfectly as swap for 16 GB main memory.

You might change the partition type of /dev/sda5 to swap.

root@edge:~# fdisk -l /dev/sda
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00023929
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          501758   976771071   488134657    5  Extended
Partition 2 does not start on physical sector boundary.
/dev/sda5       960774144   976771071     7998464   82  Linux swap / Solaris <b>(1)</b>
/dev/sda6          501760   960774143   480136192   83  Linux
Partition table entries are not in disk order

Change this partition type to Linux swap

make and enable swap

mkswap /dev/sda5
# output
Setting up swapspace version 1, size = 15624188 KiB
no label, UUID=3e021d3e-6faf-4f71-bef6-7acb1e49aa75
# activate swap
swapon /dev/sda5
echo "UUID=3e021d3e-6faf-4f71-bef6-7acb1e49aa75 none swap defaults 0 0" >> /etc/fstab (1)

add swap partition to /etc/fstab

ssd linux