Bhyve Hypervisor Freebsd ZFS

  • Get updated to at least Freebsd 11.0

    Bhyve will function on previous releases after 10.2, maybe.

    Install bhyve and bhyve grub:

    pkg install vm-bhyve grub2-bhyve

    Load kernel modules:

    add these directives into /boot/loader.conf:


    We could load the modules manually, however I’ve had issues doing this the first time, but go ahead and try, maybe it will ‘just work’ for you.

    kldload if_bridge if_tap nmdm vmm

    When that does not work, (you might not even know at this point) just reboot and the modules will be loaded at that time.

    Add the following lines to /etc/rc.conf:


    If you’ve not already setup storage, lets do it now. If you’ve not changed the default name for ZFS, it should be ‘zroot’, find out by running:

    zfs list

    On this machine the output is:

    bootpool             144M  1.72G   142M  /bootpool
    zroot               30.8G   860G    88K  /zroot
    zroot/ROOT          17.4G   860G    88K  none
    zroot/ROOT/default  17.4G   860G  17.4G  /
    zroot/tmp           40.3M   860G  40.3M  /tmp
    zroot/usr           11.6G   860G    88K  /usr
    zroot/usr/home      8.55G   860G  8.55G  /usr/home
    zroot/usr/ports     2.45G   860G  2.45G  /usr/ports
    zroot/usr/src        633M   860G   633M  /usr/src
    zroot/var           10.5M   860G    88K  /var
    zroot/var/audit       88K   860G    88K  /var/audit
    zroot/var/crash       88K   860G    88K  /var/crash
    zroot/var/log        692K   860G   692K  /var/log
    zroot/var/mail       244K   860G   244K  /var/mail
    zroot/var/tmp       9.33M   860G  9.33M  /var/tmp
    zroot/vms           1.62G   860G   500M  /vms
    zroot/vms/freebsd1  1.13G   860G  1.13G  /vms/freebsd1

    As we can see, ‘zroot’ is the pool name. !Important If the pool is named something other than ‘zroot’ , you must change the directives we did a minute ago within /etc/rc.conf to reflect the proper zpool.

    Create the storage:

    zfs create -o mountpoint=/vms zroot/vms


    vm init

    If you get the error message "$vm_enable is not enabled in /etc/rc.conf!" ect… , this is due to the modules not being loaded as described earlier, just reboot and come back. Bookmark this page or check your history.

    Now we copy the templates to the /vms directory we’ve created:

    cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/

    Creating the network bridge VMs will use:

    First lets find out what our inteface is called:


    In this case the interface is called ‘em0’


    vm switch create public
    vm switch add public em0

    Edit the template config files to use zvol instead of the IMG or iso You can use the iso, however we have ZFS for a reason don’t we.

    Templates are located where we copied them to earlier: /vms/.templates

    List all config files in the .templates directory:

    ls /vms/.templates

    Here are the default conf files listed:

    alpine.conf		centos7.conf		debian.conf		freebsd-zvol.conf	openbsd.conf		windows.conf
    centos6.conf		config.sample		default.conf		netbsd.conf		ubuntu.conf

    Have a look at the debian.conf template:


    You can edit what you wish, but know what your editing, and what it will do.

    To make use of zvol we edit both “disk0_name” & “disk0_dev” from the above to look like this:


    Time to grab an iso, lets use freebsd: Head over to and grab 11.1

    Copy the URL of the iso Running the following will snatch the image and deposit it locally.

    vm iso

    Time to create the VM: This will create a VM of freebsd named ‘freebsd11’ with a disk size of ten gigs

    vm create -s 10G freebsd11

    Now install the image we just downloaded:

    vm -f install freebsd11 FreeBSD-11.1-RELEASE-amd64-disc1.iso

    You will be asked to select [vt100] for the console type, unless you know what’s up otherwise, choose it. Now continue with installation as you would normally.

    Once install is complete, log in and play, then type:


    List all VM’s:

    vm list
    NAME            DATASTORE       LOADER      CPU    MEMORY    VNC                  AUTOSTART    STATE
    freebsd11        default         bhyveload   1      256M      -                    No           Running (1923)

    Start the VM:

    vm start freebsd11

    To destroy the VM:

    vm destroy freebsd11

    To edit the config file for specific VM:

    vm configure freebsd11

    This will open the config file in your default editor.

  • Alternatively or in addition to the above to install bhyve:

    pkg install vm-bhyve bhyve-firmware bhyve-rc-3 grub2-bhyve


    To ensure binaries built with this toolchain find appropriate versions of the necessary run-time libraries, you may want to link using


    For ports leveraging USE_GCC, USES=compiler, or USES=fortran this happens transparently.

    ===> NOTICE:

    This port is deprecated; you may wish to reconsider installing it:

    Unsupported by upstream. Use GCC 6 or newer instead… Message from vm-bhyve-1.1.8_1:

    To enable vm-bhyve, please add the following lines to /etc/rc.conf, depending on whether you are using ZFS storage or not. Please note that the directory or dataset specified should already exist.




    Then run :

    vm init

    If upgrading from 1.0 or earlier, please note that the ‘guest’ configuration option is no longer used.

    Guests that are not using UEFI boot will need either loader=“grub” or loader=“bhyveload” in their configuration in order to make sure the correct loader is used.

    Message from bhyve-rc-3:

    Configuration is done completely though rc.conf. The rc script won’t touch any devices for you (neither disk, nor tap) so you need to make sure all of those have been initialized properly.

    General setup:

    kldload vmm

    Make it persistent:

    echo "" >> /etc/sysctl.conf
    cat >> /boot/loader.conf << EOF

    Minimal example:

    cat >> /etc/rc.conf << EOF
    cloned_interfaces="tap0 bridge0"
    ifconfig tap0 create
    ifconfig bridge0 create
    service bhyve start
    tmux list-sessions
    tmux attach -t bhyve
    service bhyve status
    service bhyve stop

    Multi profile configuration example:

    cat >> /etc/rc.conf << EOF
    cloned_interfaces="tap0 tap1 bridge0"
    bhyve_profiles="virt1 virt2"
    ifconfig tap0 create
    ifconfig tap1 create
    ifconfig bridge0 create
    service bhyve start # start all
    service bhyve start virt2 # start individual
    tmux attach -t bhyve_virt1
    tmux attach -t bhyve_virt1
    service bhyve stop virt2 # stop individual
    service bhyve stop # stop all

    (by default ctrl-b d detaches from tmux).

Tmux Commands

screen and tmux

A comparison of the features (or more-so just a table of notes for accessing some of those features) for GNU screen and BSD-licensed tmux.

The formatting here is simple enough to understand (I would hope). ^ means ctrl+, so ^x is ctrl+x. M- means meta (generally left-alt or escape)+, so M-x is left-alt+x

It should be noted that this is no where near a full feature-set of either group. This - being a cheat-sheet - is just to point out the most very basic features to get you on the road.

Trust the developers and manpage writers more than me. This document is originally from 2009 when tmux was still new - since then both of these programs have had many updates and features added (not all of which have been dutifully noted here).

Action tmux screen
start a new session tmux OR
tmux new OR
tmux new-session
re-attach a detached session tmux attach OR
tmux attach-session
re-attach an attached session (detaching it from elsewhere) tmux attach -d OR
tmux attach-session -d
screen -dr
re-attach an attached session (keeping it attached elsewhere) tmux attach OR
tmux attach-session
screen -x
detach from currently attached session ^b d OR
^b :detach
^a ^d OR
^a :detach
rename-window to newname ^b , <newname> OR
^b :rename-window <newn>
^a A <newname>
list windows ^b w ^a w
list windows in chooseable menu ^a "
go to window # ^b # ^a #
go to last-active window ^b l ^a ^a
go to next window ^b n ^a n
go to previous window ^b p ^a p
see keybindings ^b ? ^a ?
list sessions ^b s OR
tmux ls OR
tmux list-sessions
screen -ls
toggle visual bell ^a ^g
create another window ^b c ^a c
exit current shell/window ^d ^d
split window/pane horizontally ^b " ^a S
split window/pane vertically ^b % ^a |
switch to other pane ^b o ^a <tab>
kill the current pane ^b x OR (logout/^D)
collapse the current pane/split (but leave processes running) ^a X
cycle location of panes ^b ^o
swap current pane with previous ^b {
swap current pane with next ^b }
show time ^b t
show numeric values of panes ^b q
toggle zoom-state of current pane (maximize/return current pane) ^b z
break the current pane out of its window (to form new window) ^b !
re-arrange current panels within same window (different layouts) ^b [space]
Kill the current window (and all panes within) ^b killw [target-window]
  • Use the same script for updating/ upgrading

    Make sure to change the versions to the latest releases:

    #!/bin/bash set -e bpcver=4.2.1 bpcxsver=0.57 rsyncbpcver=

    Scroll through the script, know what you are doing.

    Uncomment the upgrade section(s) and comment out the install section(s)

    read more
  • Again running smartctl after all is said and done:

    smartctl --all /dev/sda

    ddrescue-smartctl-after-rescue.png ddrescue-smartctl-2.png

    Yet an old drive in itself, I run the wheels off of them, and monitor regularly as anyone should.

    read more