• On Demand Offline Backups

    I have a few monthly digital magazine subscriptions that I have been getting for years. I have saved and aggregated on my hard disk. Recently, I lost a chunk of these due to my mistake at the terminal. As a precaution, I had these sync to my cloud storage space on hubic (Offer 25GB of storage space + 50 with 5 referrals). I thought I hand a second copy secured. However, as I logged in to the hubic web interface, I saw files vanishing right before my eyes. The hubic client program syncing between my computer and the cloud was sending instructions to get rid of files that were no longer on the computer. By the time I could log in via SSH and terminate the client program, it was too late. Unlike Dropbox, hubic does not keep deleted files to undelete. Years of aggregated magazine subscriptions lost in minutes.

    This made me see the necessity to have offline backups.

    One easy way was to set cron tasks to periodically copy certain folders to an external drive via rsync. However, leaving an external drive connected does turns this "offline" approach into "online". I felt, it would be ideal if I could plug in my drive into my NAS whenever, I wanted and this would cause a backup job to start running. I looked around and discovered that I could use udev, but reading more, I decided I did not want to dive into udev for this.

    I finally settled on an idea, where I plugin my hard disk into the NAS whenever, I wanted to back up and that night, the selected folders would back up to the hard drive. The next morning, I could unplug the disk and voila, I have my little offline backup system.

    I set about implementing this using a shell script that would run every night and check, if my external USB backup, is plugged in or not. If found, it would initiate a backup sequence. Here is the shell script that I used -

    #! /bin/bash
    echo "===== Running at $now ====="
    if [ -b /dev/disk/by-uuid/"$PART_ID" ]; then
            echo "Partition Found!"
            mkdir ~/$PART_ID
            mount $PART_ID
            if [ $? -eq 0 ]; then
                    echo "Mounted"
                    # Run backups
                    rsync -r "/home/bharath/hubic/Datasheets" ~/$PART_ID/.
                    rsync -r "/home/bharath/hubic/Harvard Business Review" ~/$PART_ID/.
                    rsync -r "/home/bharath/hubic/IEEE Computer Magazine" ~/$PART_ID/.
                    rsync -r "/home/bharath/hubic/Linux Journal" ~/$PART_ID/.
                    rsync -r "/home/bharath/hubic/MagPi" ~/$PART_ID/.
                    rsync -r "/home/bharath/hubic/Make Magazine" ~/$PART_ID/.
                    rsync -r "/home/bharath/Photos" ~/$PART_ID/.
                    rsync -rl "/home/bharath/Datasets" ~/$PART_ID/.
                    umount $PART_ID
                    pushb "Offline backup done."
                    if [ $? -eq 0 ]; then
                            echo "Unmounted"
                            rm -Rf ~/$PART_ID
                            echo "===== Done at $now ====="
            echo ":-("

    Here is the gist containing the code.

    To have this script work, I had to add a line allowing users to mount the disk in the /etc/fstab file.

    UUID=926fc38e-bc7e-467a-953e-25c6555939a0       /home/bharath/926fe38e-bc7e-467a-959e-25c6535939f0      ext4    noauto,noatime,nodiratime,relatime,user 0       1

    Apart from this, I had to set the user bharath (me!) rwx permissions on the entire disk. A big problem with doing this is that If, I move the disk to a different computer - one that does not have the user bharath (me!) created with the user id of 1000, someone else, who has UID=1000 would have the rwx permissions. This is something, I would have to live with till I figure a way around this or write my own portable journaling file system.

    To perform a nightly check, I would have to add something like the following line to my crontab -

    0 4 * * * /home/bharath/do_offline_backup.sh >> offline_backup.log

    On a side note, comparing hubic to Dropbox, I see hubic having a huge price advantage -

    hubic Dropbox AWS S3 (Oregon)
    €10/year (100 GB) $100/year (1TB) $370/year (1TB)
    €50/year (10 TB) + Data transfer cost

    * 2016 October Prices

    And hubic servers are hosted in France, we can reasonably hope that this in out of reach of the various 3 letter government agencies of the US.

    The downside of hubic are -

    1. Speed - it is slightly slower than dropbox.
    2. UI - not as great a UI as dropbox.
    3. Versioning - no versioning or undelete.
    4. Dropbox paper - no on website editors with hubic.

    If all you want to do is have an online cloud backup, this is the service for you.

  • Hardcoding Passwords and git

    Recently I was working on building myself a script to download my securities transactions from Robinhood and save it as a CSV. I found an unofficial python library that was reverse engineered from the app. As I started building the script, I realized that I could not host it on git. I had hardcoded my credentials in the code.

  • Retrieving Historical Stock prices.

    Yahoo Open Data Tables is probably the last free source of financial information available after Google shut down it's finance API a few years back. In a recent project to analyze my stocks trades, I decided to cache 10 year historical prices form the Yahoo Open Data Tables.

  • pip for Cygwin

    I have been using cygwin for a while. To say the least, I get to do X11 forwarding on windows. Windows 10 now has an experimental feature called Bash for Windows. I think this was released along with the Windows 10 Anniversary update. However, I have not succeeded in getting X11 to run on it.

  • Terminal Malfunction

    Occasionally, something I do at the terminal causes the terminal to misbehave at the end of the output. This usually happens when I display a binary file. For example -

subscribe via RSS