ZFS snapshot on FreeBSD – Part 2

In my previous post I presented how to do live backups of a running FreeBSD server with ZFS snapshot, today I’m doing the follow up: how to increment that snapshot.

If you have a big amount of data initial zfs send|receive can take a lot of time, many hours, or days. With incremental zfs send|receive we only need to send the difference. The difference between original snapshot and second snapshot.

  • In my case I’m using a USB drive, let’s attach the drive and import it.

sudo zpool import zbackup

  • Create a new snapshot
    sudo zfs snapshot -r zroot@20180420
  • Send the incremental snapshot
    sudo zfs send -vRi zroot@20180411 zroot@20180420 | sudo zfs receive -vu zbackup/zroot_20180411

This didn’t work
receiving incremental stream of zroot/usr/home@20180420 into zbackup/zroot_20180411/usr/home@20180420
cannot receive incremental stream: destination zbackup/zroot_20180411/usr/home has been modified
since most recent snapshot
warning: cannot send ‘zroot/usr/home@20180420’: signal received

Ok, I remember I was doing this before, and I used -F with send. Let’s see the manual, this should be fine

sudo zfs send -vRi zroot@20180411 zroot@20180420 | sudo zfs receive -vuF zbackup/zroot_20180411
send from @20180411 to zroot@20180420
send from @20180411 to zroot/music@20180420
send from @20180411 to zroot/usr@20180420
send from @20180411 to zroot/usr/ports@20180420
send from @20180411 to zroot/usr/src@20180420
send from @20180411 to zroot/usr/home@20180420 estimated size is 158M
send from @20180411 to zroot/share@20180420 estimated size is 21.7M
send from @20180411 to zroot/tmp@20180420 estimated size is 1.04M
send from @20180411 to zroot/film@20180420
send from @20180411 to zroot/files@20180420 estimated size is 246M
send from @20180411 to zroot/ROOT@20180420
send from @20180411 to zroot/ROOT/default@20180420 estimated size is 2.76G
send from @20180411 to zroot/var@20180420
send from @20180411 to zroot/var/mail@20180420 estimated size is 233K
send from @20180411 to zroot/var/tmp@20180420 estimated size is 103K
send from @20180411 to zroot/var/log@20180420 estimated size is 3.07M
send from @20180411 to zroot/var/audit@20180420
send from @20180411 to zroot/var/crash@20180420
send from @20180411 to zroot/ezjail@20180420 estimated size is 618M
send from @20180411 to zroot/ezjail/newjail@20180420 estimated size is 2K
send from @20180411 to zroot/ezjail/basejail@20180420
total estimated size is 3.79G
TIME SENT SNAPSHOT
receiving incremental stream of zroot@20180420 into zbackup/zroot_20180411@20180420
TIME SENT SNAPSHOT
snap zbackup/zroot_20180411@20180420 already exists; ignoring
TIME SENT SNAPSHOT
received 0B stream in 1 seconds (0B/sec)
TIME SENT SNAPSHOT
TIME SENT SNAPSHOT
receiving incremental stream of zroot/music@20180420 into zbackup/zroot_20180411/music@20180420
TIME SENT SNAPSHOT
snap zbackup/zroot_20180411/music@20180420 already exists; ignoring
received 0B stream in 1 seconds (0B/sec)
receiving incremental stream of zroot/usr@20180420 into zbackup/zroot_20180411/usr@20180420
snap zbackup/zroot_20180411/usr@20180420 already exists; ignoring
received 0B stream in 1 seconds (0B/sec)
receiving incremental stream of zroot/usr/ports@20180420 into zbackup/zroot_20180411/usr/ports@20180420
snap zbackup/zroot_20180411/usr/ports@20180420 already exists; ignoring
received 0B stream in 1 seconds (0B/sec)
receiving incremental stream of zroot/usr/src@20180420 into zbackup/zroot_20180411/usr/src@20180420
snap zbackup/zroot_20180411/usr/src@20180420 already exists; ignoring
received 0B stream in 1 seconds (0B/sec)
receiving incremental stream of zroot/usr/home@20180420 into zbackup/zroot_20180411/usr/home@20180420
18:42:30 114M zroot/usr/home@20180420
TIME SENT SNAPSHOT
18:42:31 158M zroot/share@20180420
18:42:32 158M zroot/share@20180420
18:42:33 158M zroot/share@20180420
received 158MB stream in 5 seconds (31.6MB/sec)
receiving incremental stream of zroot/share@20180420 into zbackup/zroot_20180411/share@20180420
TIME SENT SNAPSHOT
received 21.7MB stream in 1 seconds (21.7MB/sec)
receiving incremental stream of zroot/tmp@20180420 into zbackup/zroot_20180411/tmp@20180420
TIME SENT SNAPSHOT
TIME SENT SNAPSHOT
received 1.40MB stream in 1 seconds (1.40MB/sec)
receiving incremental stream of zroot/film@20180420 into zbackup/zroot_20180411/film@20180420
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/files@20180420 into zbackup/zroot_20180411/files@20180420
18:42:36 181M zroot/files@20180420
TIME SENT SNAPSHOT
TIME SENT SNAPSHOT
18:42:38 427M zroot/ROOT/default@20180420
.
.
18:42:44 427M zroot/ROOT/default@20180420
received 246MB stream in 9 seconds (27.3MB/sec)
receiving incremental stream of zroot/ROOT@20180420 into zbackup/zroot_20180411/ROOT@20180420
18:42:45 427M zroot/ROOT/default@20180420
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/ROOT/default@20180420 into zbackup/zroot_20180411/ROOT/default@20180420
18:42:46 525M zroot/ROOT/default@20180420
18:42:47 595M zroot/ROOT/default@20180420
.
.
.
18:44:09 3.31G zroot/ROOT/default@20180420
18:44:10 3.31G zroot/ROOT/default@20180420
TIME SENT SNAPSHOT
TIME SENT SNAPSHOT
received 2.90GB stream in 87 seconds (34.1MB/sec)
receiving incremental stream of zroot/var@20180420 into zbackup/zroot_20180411/var@20180420
18:44:12 3.32G zroot/var/mail@20180420
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/var/mail@20180420 into zbackup/zroot_20180411/var/mail@20180420
TIME SENT SNAPSHOT
received 213KB stream in 1 seconds (213KB/sec)
receiving incremental stream of zroot/var/tmp@20180420 into zbackup/zroot_20180411/var/tmp@20180420
TIME SENT SNAPSHOT
received 153KB stream in 1 seconds (153KB/sec)
receiving incremental stream of zroot/var/log@20180420 into zbackup/zroot_20180411/var/log@20180420
TIME SENT SNAPSHOT
TIME SENT SNAPSHOT
TIME SENT SNAPSHOT
received 3.19MB stream in 1 seconds (3.19MB/sec)
receiving incremental stream of zroot/var/audit@20180420 into zbackup/zroot_20180411/var/audit@20180420
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/var/crash@20180420 into zbackup/zroot_20180411/var/crash@20180420
18:44:14 3.32G zroot/ezjail@20180420
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/ezjail@20180420 into zbackup/zroot_20180411/ezjail@20180420
18:44:15 3.40G zroot/ezjail@20180420
.
.
18:44:31 3.92G zroot/ezjail@20180420
TIME SENT SNAPSHOT
TIME SENT SNAPSHOT
load: 1.18 cmd: zfs 1407 [tx->tx_sync_done_cv] 145.47r 0.00u 3.61s 0% 3116k
received 618MB stream in 110 seconds (5.62MB/sec)
receiving incremental stream of zroot/ezjail/newjail@20180420 into zbackup/zroot_20180411/ezjail/newjail@20180420
received 624B stream in 1 seconds (624B/sec)
receiving incremental stream of zroot/ezjail/basejail@20180420 into zbackup/zroot_20180411/ezjail/basejail@20180420
received 312B stream in 1 seconds (312B/sec)

If nothing happens for awhile, on FreeBSD we can press ctrl+t, which will tell us without interrupting our command execution. This won’t work on Linux. See the bold line above load: 1.18 cmd: zfs 1407 [tx->tx_sync_done_cv] 145.47r 0.00u 3.61s 0% 3116k

  • After finishes, let’s check if the snapshot is there.
    zfs list -t snapshot
  • Now we can remove the original snapshot from the server. Creating the snapshot can be done with a single command, but we can’t remove it recursively. Let’s do it with a for loop
    • actually bot yet, we need to keep three snapshot in order to be able to do incremental snapshots, so before you remove any snapshots be sure to keep the last 3
      zfs list -H -r -t snapshot -o name zroot|grep 20180411
      for i in `sudo zfs list -H -r -t snapshot -o name zroot|grep 20180411`; do echo sudo zfs destroy -vn $i; done

      Check the output meticulously! If we like what we see remove the “echo” and the “n”
  • Let’s see how our snapshots look
    zfs list -H -r -t snapshot -o name,creation zroot
    zfs list -H -r -t snapshot -o name,creation zbackup
  • Ok, it is time to remove the USB drive
    sudo zpool export zbackup
    cannot unmount '/usr/home': Device busy

    Wait, what’s happening here?

I did a mount and I saw that I have two mounts from zbackup, one of them happens to be /usr/home. I did cd /, then zpool export zbackup, it worked.

My initial zfs send|receive took a little more than 16 hours, this took 5 minutes.

Task for next time is to find out why did those two mount points got mounted.

 

Update.

I realized there are a few more things to be added here.
One – you must keep at least three snapshots to be able to do an incremental snapshot send, for example keep zroot@20180411, zroot@20180412, zroot@20180413.
Two – recursive snapshot removal can be done easier than a for loop: zfs destroy -v[n]r zroot@20180411

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.