[kwlug-disc] btrfs/zfs for backups

[B.S.]

Hold on ... how have you gone from file replication not working to 
entire server and backup infrastructure replacement?

Let alone with snapshotting entire filesystems rather than file copying 
- I must be missing something. If you have to restore, would it have to 
come back via wi-fi too? (What's your downtime, including restore time, 
tolerance?)

Backups and files of the size you are talking about means you're taking 
more than just data files (spreadsheets and documents), but system drive 
images in case of failure, too. I wouldn't want to be slinging them 
around a network, especially wireless. 2 bad bits in GB files may mean 
complete resends, while with files you may get 100's of successful 
copies in between, and a bad bit should mean only one smaller file need 
be re-sent. (One of the reasons / advantages for incremental backups - 
irritating and useless as they are.)

You might try rsync first, even on an intermediary linux box that sucks 
from one server and pushes to the other. At least only changed files 
should go over - you may be able to replace your DFS infrastructure with 
this and keep the rest the same. (I do this nightly, gb's worth, 
including my mondoarchive generated system partition image iso's - GB 
WIRED!)

rsync also has the ability to only send changed bits within a file (but 
only via demons) - I don't recall whether a demon only on the win server 
side will 'just work', but it shouldn't be too hard to try with cygwin. 
(i.e. Avoid the problematic rsync windows client problem.) Even taking 
your laptop over and trying an rsync from it to the demon, pushing 
(rather than pulling, or vice versa), relaying to your real destination, 
may give you a sense of viability.

For a windows equivalent, look at robocopy.

For that matter, the baby box could map to the server samba shares (C$, 
D$, {x}$), and the baby box could run the daemon, being Linux and Linux 
having viable rsync implementations?

For sheer safety and speed (of users during the process), you might want 
to put a linux box beside each, then replicate between them. This way 
you could get a copy off the server to a baby box beside it at wire 
speed overnight, then let the two baby boxes replicate over the wi-fi 
afterwards - at least you'll know you have one good backup nearby, not 
half sent and possibly mis-sent over the air. And it won't matter how 
long the OTA process takes, as long as it's less than 24 hours, and the 
servers themselves won't be burdened (decreasing user response time) in 
the mean time.

I get the advantages of the large backups, aside from "here's the 
backup", you are likely also getting archiving and file history 
advantages. Perhaps it's time to bite the bullet and sink into the whole 
Amanda (and NFS?) ecosystem? SFU? Even a BackupExec image dumped off to 
a nearby baby brother linux box, which then Amanda's things off in your 
desired history / archive / ageing strategy? (But doesn't that bring a 
double restore / unpack / unencrypt burden avoided by direct native 
solutions?)

On 12/03/2014 02:04 PM, Paul Nijjar wrote:
 > Okay. You and others have convinced me to take another look at
 > robocopy and xcopy. Because there is so much data, it is important not
 > to copy files that don't need copying.

But you shouldn't have large files to sling around in the first place. 
Except perhaps for system partition images, and even they are only 
useful until the next one is generated. (Assuming one good copy is 
burped down the line.) i.e. Storage at wire speed on a box beside the 
beast keeps you safe while sending less over the wi-fi, or at least 
letting everything else be more robust / safer for the duration? [System 
partitions being atomic backup beasties, preferably. Shadow copy snapshots?]

Even SQL and Exchange you back up transaction logs / dumps (text files) 
in small chunks and roll things backwards or forwards at need. The thing 
is to reduce as much as you can the large binary backup blobs.

One of the problems with snapshots is no matter what you do, they always 
want up to the minute before it went down, and with a binary blob backup 
you're always limited to up to 24 hours ago - *IF* all the backup 
processes ran perfectly that night. Thus the constant siphoning off of 
dumps and transaction logs to let you get back to that last minute. And 
inter-server/service replication, which takes the need for large binary 
backup blobs on these off the table.


On 12/02/2014 08:44 PM, Paul Nijjar wrote:
> I am officially unhappy with Microsoft DFS replication. Earlier this
> year I set up a couple of Windows Server 2008 R2 boxes to synchronize
> big backup files over a wireless link, and it does not work well at
> all. So now I am re-evaluating, and once again thinking of using some
> Free Software filesystems for my job.
>
> I was thinking of scrapping the DFS replication and trying to use
> rsync on Windows instead, but as far as I can tell all binaries of
> rsync on 64-bit Windows are terrible.
>
> My inclination is to go with Debian/Ubuntu fileservers that
> synchronize backup files (one way). Once upon a time somebody (I think
> it was Chris Irwin) got me all excited by talking about btrfs send and
> receive, that would allow you to send snapshots of incremental changes
> to remote hosts via SSH. That sounds exciting, but The Internet (or at
> least this StackExchange post:
> http://serverfault.com/questions/285878/is-btrfs-production-ready ) it
> seems that I should not consider this, and should go with ZFS instead
> (which also has this functionality).
>
> So here are my options:
> - Try btrfs on Debian/Ubuntu and hope it is mature enough to work for
>    my use case
> - Try ZFS on Linux the way Lori does
> - Try ZFS on FreeBSD or some other OS where it is native
> - Find some way to get large, effective filesync working on the
>    Windows servers I have already built (ideally with FLOSS)
>
> Here is some information about the infrastructure:
> - The fileserver will consist of a bunch of Samba shares
> - Symantec BackupExec (yes, I know) will write big backup files to
>    these shares. Most of the files are 4GB large, but there are a few
>    files that are almost 100GB large.
> - The two servers are connected via a wireless link that effectively
>    runs at 100Mbit
> - The backup storage media are Western digital Green drives (sorry
>    Cedric)
> - The servers themselves are nothing special: 64-bit intel
>    workstations with 2-4GB of RAM.
> - These are backup files, so data integrity is important
> - We can assume there are 100s of GB of data backed up each week,
>    although I am not sure whether this means hundreds of files are
>    changing each week. (This could be the case; BackupExec has a habit
>    of doing things in the most inconvenient way possible.)
>
> I am interested in hearing about how well btrfs works for the btrfs
> send/receive scenario I am thinking about, and any advice
> strengthening/contradicting the StackExchange opinion. If people are
> using ZFS (in particular ZFS on Linux) with zfs send/receive in this
> manner then I am interested in that information as well. If people
> have other options (such as an effective rsync option for Windows 64
> bit) then feel free to chime in. I am more interested in experiences
> than speculation.