Taming the Drobos

I’ve been working on a super challenging technical project for months now. I’ve wanted to tell you about it but I have run into so many barriers that I was unable to overcome, there wasn’t a good story to tell. I still have not conquered my original task but I’ve had some partial success so I want to do a mini celebration about that.

The Problem to be Solved

Some of this story you may heard before but I want to bring everyone up to speed. A while ago we bought a new Drobo 5N to replace our aging (and never that good in the first place) Drobo FS. The 5N is a dream, with its SSD for caching, it’s super fast and makes a terrific place to store giant and small files alike. I set up a Hazel script to automatically pull over my audio file so my internal MacBook Pro drive doesn’t get filled up any more. Steve stores his giant Final Cut Pro X libraries over on the 5N when he’s done with them. It’s awesome.

Drobos use RAID-like proprietary tools to protect our files from a drive failure, so we do have some fault tolerance but this isn’t technically a backup since our data is only in one place.

That’s when we got the idea to use the old Drobo FS as a backup to the new Drobo 5N. Let’s just start calling them old and new Drobo for simplicity. We got a copy of Chronosync from Econ Technologies and after a bit of work we successfully got it to sync our files from the new to the old Drobo. We have 4TB of data on the new Drobo so that took a while to get started but after a few weeks it was running fine. I even did a tutorial I called How to Back Up from One Drobo to Another.

For some unknown reason that will leave me lying awake at night for the rest of my life, something went terribly wrong. The symptom we first noticed was that the old Drobo was full. No problem, Steve bought two more 2TB drives and popped them in the empty drive bays on each Drobo and we were back in business. Two days later the old Drobo was full again. Uh oh.

I’ll spare you the weeks and weeks of trying to figure out what had gone wrong and tell you that the culprit is Steve’s Final Cut Pro X libraries. We’re not sure why but they grow out of control when synced to the new Drobo. I want to say that Tom from Econ Technologies has been relentless in trying to help me figure this out. He’s worked with his engineers and given me a plethora of solutions none of which have worked. Over time I started eliminating variables and I was able to eliminate Chronosync from the root cause, but even after I let Tom off the hook, he still kept sending me ideas. Near as we can figure it’s something about symbolic vs. hard links that’s giving it fits.

Dorothy, aka @maclurker, goes to the gym with me every other day and has had to listen to me blather on about this problem for months and months. Luckily she likes this technical blather and she came up with an interesting thought. My syncing process uses a Mac mini in between the two Drobos, and maybe it’s the fact that the Mac is doing some interpretation of these files in between that’s giving it fits. She suggested I see if I can figure out a way to go from one Drobo directly to the other without ever using a Mac in between. This would be much faster because the two Drobos are on a gigabit switch and it would be a direct one way copy. Right now the Mac mini is on wifi and the files have to go from new Drobo to Mac mini to old Drobo.

And now I’ve set the stage for a fun geeky adventure. I want to say here that I would have never undertaken such a crazy thing if it weren’t for Bart’s Taming the Terminal series. I’m sure as you listened to that series, you figured out that I wasn’t following every step he told me about and you clearly noticed that I didn’t remember much from lesson to lesson. But what Taming the Terminal did for me was let me know that I could do these things, and that his written tutorials would be there when I needed to execute.

rsync Between Two Drobos

Underneath the hood of probably all syncing programs on the Mac is a unix tool called rsync. I figured if I could get rsync running on both Drobos, I could push files from new to old Drobo without using the Mac mini.

Drobo doesn’t specifically support 3rd party apps on their NAS devices, but they begrudgingly have something called DroboApps. It’s turned on by default on the new Drobo but on the old one it took some effort to find and enable it.

I found a site called Drobo Ports where they had ported a bunch of linux tools to run on the Drobo which was awesome. The downside was that they sorta assumed that you had a clue what you were doing in linux. While I claim to have tamed the Terminal, we all know that I’m a low novice if we round up! What I do have is a darn good bit of determination combined with some good skill at Google searches, and those two things combined helped me succeed.

If I’m going to talk to the Drobos directly, I can’t be using the Finder, right? I need to be able to secure shell, or SSH into them. I downloaded two zipped tar files (think of tar as a ball of files in a container) with the separate executables for OpenSSH for each Drobo. I followed the instructions to drop those two tar files into the DroboApps folder and rebooted and then I was able to ssh into both Drobos. Sweet. Poked around a bit, learned the file structure, good to go.

Next up I copied the rsync tar files over. These would require me to specifically install them using the command /usr/bin/DroboApps.sh install. But that didn’t work because I didn’t have permissions to install software. No problem, I’ll just use sudo, or super user do, because that temporarily raises my permissions. But guess what? The installation of linux on the Drobos doesn’t have sudo! No problem…I’ll just install sudo. wait…um, how do I install sudo if I don’t have sudo?

As a joke I posted that question to Twitter and my friend Steve Brady responded with a link to an article that explained how to get into root so you could install sudo. Cool beans. Now I’ve got sudo, I can install rsync. Goes off without a hitch, even got the notice that said rsync is installed and running.

I figured I should try a test first, so on the new Drobo I created two directories: test_dir1 and test_dir2. I opened test_dir1 and created two dummy files by using the touch command. If you simply type touch and a file name, it causes it to exist. I know, kind of an existential concept. Anyway, now I’ve got 2 files in test_dir1 and I want to rsync those files into test_dir2. After looking up the syntax to do that I tried it…and it failed.

It turns out rsync, while running, was not in my PATH.

Remember when Bart explained PATH to us? I do, but I didn’t understand it at the time. I also didn’t understand it when my friend Adam explained it to me a few years before. I didn’t understand it when my friend Ryan explained it to me before that. Or when Mario and even earlier Chris explained it to me before that. It simply never sunk in.

Now that I actually needed to understand it, I think I finally “got” it. I’m going to use an analogy to give my understanding of PATH. This may be off base but luckily I am confident the NosillaCastaways will correct me if I’m wrong. Imagine if on your Mac, the only way applications could run was if they were in your Applications folder, and only official apps were allowed to go in the Applications folder. If you had a 3rd party app, you could do one of two things: you could either specify the exact location of the application on your drive, OR you could tell OSX where to look for 3rd party apps. That’s what putting it in PATH means on Linux.

Using what Bart taught us I could see where PATH was going to look so I knew where to put my apps. I did some googling on how to make this work on a Drobo and found some lovely instructions that explained how to put the entire DroboApps section into the PATH. This means that whatever I put in DroboApps in the future will run without any extra work. Even better, it actually worked. I did my little test sync within the new Drobo and then within the old Drobo so I knew rsync was working on both devices.

But that’s really only half the problem, we know it’s running on both Drobos but we want to be able to run rsync from one Drobo to the other Drobo. No problem, that should work, right?

Of course it didn’t. When I told the new Drobo to rysnc over to the old Drobo, it said that rsync wasn’t running on the old Drobo! Well dagnabbit, we know it’s running, we just used it. More of the googles and I figured out that when you remote into a device, it doesn’t run the stuff you just painstakingly put in your PATH!

The next suggestion I found said that if I just created a symbolic link that would solve everything. I have no idea what this means or why it would work. Basically you type ln for link, -s to specify a symbolic link, then you tell it where the file is and where you want the link to go. I don’t get why this would help because if I put this symbolic link in the right place in my PATH but ssh never sees my PATH, why would it work? Either my instincts were right or I did it wrong, but in any case it didn’t work.

I then found out that the Drobo FS (the old one) doesn’t follow symlinks by default anyway! Found someone who suggested editing a file called /etc/.backup.smb.conf and adding a line that says to follow symlinks amongst other suggestions but that didn’t work either.

Ok, time for more of the googles. I found a lot of people asking about this and in a few spots I found a suggestion to define the whole darn path to rsync every time you run it. This actually worked!

From the new Drobo, I can enter simply rsync, because I’m ssh’d in and I have it in my PATH. From there I define the directory I want to sync, followed by a trailing backslash because I don’t want it to copy the directory just what’s in it. Next is -a which means archive, -v for verbose so I can watch it working. Now for a tricky bit: –rsync-path= followed by the full path to rsync on the old Drobo. With me so far? Now follow that with -e, which means to use the remote shell command to execute. Next we have to tell it what user name to use on the old Drobo so we enter “ssh -l username” in quotes. No idea why it’s in quotes. And finally the IP address of the old Drobo followed by a colon and the full path where I want the synced stuff to go:

$ rsync test_dir1/ -av --rsync-path=/mnt/DroboFS/Shares/DroboApps/rsync/bin/rsync -e "ssh -l spsheridan" 10.0.1.202:/mnt/DroboFS/Shares/al_backup_old/test_dir1

So in only four days using about 127 websites, everything Bart ever taught me, Dorothy’s inspiration, a little bit of Twitter and a lot of Google, I was able to transfer two empty files from the new Drobo 5N to the old Drobo FS. It was glorious.

And then I tried it on some of my real files and it failed.

UPDATE: I figured out eventually that I needed to change ownership on the files in question to the user account instead of root, and after that it worked!!!