| GeomGate |
geom gatate allows mounting a raw filesystem on a remote server. when used in combination with gmirror, a local filesystem can be mirrored to a remote server.
This sounds awesome and works nicely, but in my experience has some serious limitations which make automated failover in event of a hardware failure impossible. If you know a solution for any of these problems, please let me know!
gmirror commands while the remote ggated box is down
will hang everything on the local system.# ggate ggated_enable="YES"
############################################################## ### Jail Configuration ####################################### ############################################################## ifconfig_dc0_alias1="inet 192.168.1.99 netmask 0xffffffff" ifconfig_dc0_alias0="inet 192.168.1.98 netmask 0xffffffff" jail_enable="YES" jail_set_hostname_allow="NO" jail_socket_unixiproute_only="YES" jail_sysvipc_allow="NO" jail_list="geektank core" # active jail server runs ggatec ggatec_enable="YES" jail_geektank_ip="192.168.1.99" jail_geektank_hostname="geektank.subaudi.net" jail_geektank_rootdir="/mnt/jails/geektank" jail_geektank_exec="/bin/sh /etc/rc" jail_geektank_devfs_enable="YES" jail_geektank_procfs_enable="YES" jail_geektank_mount_enable="NO" jail_core_ip="192.168.1.98" jail_core_hostname="core.subaudi.net" jail_core_rootdir="/mnt/jails/core" jail_core_exec="/bin/sh /etc/rc" jail_core_devfs_enable="YES" jail_core_procfs_enable="YES" jail_core_mount_enable="NO"
| grep ggate; do sleep 180; done |
Here is another link to get some information on failover:
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=262144
kern.ipc.maxsockbuf=1048576
serverB# ggated -R 262144 -S 262144
serverA# ggatec create -R 262144 -S 262144 -o rw 10.10.10.3 /dev/da0s1g
# OR
serverA# ggatec create -o rw serverB /dev/ad2s1d
ggate0
serverA# mount -t ufs /dev/ggate0 /export/mirror02b
To start, I had a local 300G drive mounted locally on serverA, and an identical 300G drive mounted remotely on serverB. I used ggated on serverB and ggatec on serverA as described above. Then I performed the instructions below on serverA.
Note that you want to insert the remote disk into the mirror first. The disk in the mirror with the highest priority gets used first if you set the balance algorithm to 'prefer'--it seems like a good idea to make sure that's the local disk... The first item inserted in the mirror automatically gets a priority of 0. There is a documented limitation of gmirror--you can't change the priority of a component after it's added. If you add the local disk first and the filesystem is live (being changed), you'll need to do a full sync, then remove the local disk, and then add the local disk back with a higher priority and complete another full sync--which can take some time over the network. I used smaller disks for this project to limit the upper boundary on network sync times.
# local drive is: /dev/ad2s1d
# remote drive is: /dev/ad1s1 on serverB mounted to /dev/ggate0
# create ggate device
ggatec create -o rw serverB /dev/ad1s1
# Make ABSOLUTELY CERTAIN that you don't have any pre-existing GEOM labels
gmirror clear /dev/ad2s1d
gmirror clear /dev/ggate0
# zero out the remote disk
dd if=/dev/zero of=/dev/ggate0 bs=512 count=79
# place a GEOM mirror label onto remote disk
gmirror label -v -b prefer gm0 /dev/ggate0
# activate GEOM mirror kernel layer - makes /dev/mirror/gm0 available
gmirror load
# place a PC MBR onto the remote disk - single FreeBSD slice /dev/mirror/gm0s1 covering the whole disk
fdisk -v -B -I /dev/mirror/gm0
# place a BSD disklabel onto /dev/mirror/gm0s1
# (NOTICE: figure out what partitions you want with "bsdlabel /dev/ad0" before)
# (NOTICE: start "a" partition at offset 16, "c" partition at offset 0)
bsdlabel -w -B /dev/mirror/gm0s1 # initialize
bsdlabel -e /dev/mirror/gm0s1 # create custom partitions
# manually copy filesystem data from local to to remote disk
# (same procedure for partitions "g", etc)
newfs -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt/mirror02_remote
rsync -rav /export/mirror02_local /mnt/mirror02_remote
# unmount local drive
umount /export/mirror02_local
# adjust new system configuration for GEOM mirror based setup
vi /etc/fstab
# change to using: /dev/mirror/gm0 instead of /export/mirror02_local
# make sure the local disk is treated as a really fresh one
# (also not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad2s1d bs=512 count=79
# switch GEOM mirror to auto-synchronization and add local disk
# (local disk is now immediately synchronized with the remote disk content)
# set priority to 5 to prefer the local disk
gmirror configure -a gm0
gmirror insert -p 5 gm0 /dev/ad2s1d
# monitor gmirror output
gmirror list
# enable gmirror at startup
echo 'geom_mirror_load="YES"' >>/boot/loader.conf
# reboot into the final two-disk GEOM mirror setup
# (now actually boots with the MBR and boot stages on local disk
# as it was synchronized from remote disk)
shutdown -r now
Here is my FreeBSD ggated startup script. This goes in /usr/local/etc/rc.d, and must be enabled in rc.conf. For more info, see the FreeBSD handbook entry on rcng.
#!/bin/sh
# PROVIDE: ggatec
[ -z "${ggatec_enable}" ] && ggatec_enable="NO"
name=ggatec
. /etc/rc.subr
rcvar=`set_rcvar`
start_cmd="ggatec create -t 180 -R 262144 -S 262144 -o rw vicious /dev/ad1"
stop_cmd="/usr/bin/killall ggatec"
load_rc_config $name
run_rc_command $*
#!/bin/sh
# PROVIDE: ggated
[ -z "${ggated_enable}" ] && ggated_enable="NO"
name=ggated
. /etc/rc.subr
rcvar=`set_rcvar`
start_cmd="/sbin/ggated -R 262144 -S 262144"
stop_cmd="/usr/bin/killall ggated"
load_rc_config $name
run_rc_command $*
