Jekyll2018-04-11T22:28:47-05:00https://ratnavali.net/weblog/KaryashalaWrite an awesome description for your new site here. You can edit this line in _config.yml. It will appear in your document head meta (for Google search results) and in your feed.xml site description.Reading a part of a file2017-07-03T00:00:00-05:002017-07-03T00:00:00-05:00https://ratnavali.net/weblog/reading_a_part_of_a_file<p>Recently I had a problem with a very large file I downloaded and wanted to have an alternative option to re-downloading it. The checksum of the file with the problem was incorrect. So I decided to look for ways to compute the MD5 checksums for parts of the file.</p>
<h2 id="how-to-read-a-part-of-the-file">How to read a part of the file?</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dd if=<filename> skip=<number of blocks> bs=<block size> count=<no of blocks>
</code></pre></div></div>
<p>I found this solution on <a href="https://stackoverflow.com/a/219188/482176">stackoverflow.com</a></p>
<p>So, to read a file from byte 1025 to byte 4096,</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dd if=lorem_ipsum.txt skip=1024 bs=1 count=3072
</code></pre></div></div>
<h2 id="now-for-the-md5sum">Now for the md5sum…</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dd if=lorem_ipsum.txt skip=1024 bs=1 count=3072 | md5sum
</code></pre></div></div>Bharath Bhushan LohrayRecently I had a problem with a very large file I downloaded and wanted to have an alternative option to re-downloading it. The checksum of the file with the problem was incorrect. So I decided to look for ways to compute the MD5 checksums for parts of the file.Counting MATLAB Figures2017-07-03T00:00:00-05:002017-07-03T00:00:00-05:00https://ratnavali.net/weblog/counting_MATLAB_fugures<p>I needed to write a few scatter plot images to disk from my MATLAB scripts. I was creating a few hidden figures for this purpose.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>figure('Visible','off');
</code></pre></div></div>
<p>The problem with this script was that it soon racked up a lot of hidden figures. These hidden figures consume memory and slow down the computer. So I started looking for a way to count the number of figures that are open in MATLAB and found <a href="https://stackoverflow.com/questions/4540604/how-do-i-get-the-handles-of-all-open-figures-in-matlab">this</a> answer on stackoverflow.com.</p>
<p>Accoding to the post, the solution that would work is -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>figHandles = get(groot, 'Children');
length(figHandles)
</code></pre></div></div>
<p>An alterate way to prevent the problem from occuring is by reusing the figures. I did this by -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if (~exist('fid2_fig','var'))
fid2_fig=figure('Visible','off');
end
...
...
...
if (exist('fid2_fig','var'))
close(fid2_fig);
clearvars('fid2_fig');
end
</code></pre></div></div>
<p>This keep track of the figures spawned by assigning the figure handles to a variable. If the figure handle exists and is valid in the second loop, there would be no need to spawn an additional figure.</p>
<p>Now, if a figure (visible) is closed, the variable that holds the figure handle does not vanish from the workspace.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fid2_fig =
handle to deleted Figure
</code></pre></div></div>
<p>One may check if a figure handle is actually valid by -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if (~isvaild(fid2_fig))
fid2_fig=figure();
end
</code></pre></div></div>Bharath Bhushan LohrayI needed to write a few scatter plot images to disk from my MATLAB scripts. I was creating a few hidden figures for this purpose.Securing with Port Knocking2017-04-30T00:00:00-05:002017-04-30T00:00:00-05:00https://ratnavali.net/weblog/securing_with_port_knocking<p>Recently I discovered that my log files started growing. The biggest among them was <code class="highlighter-rouge">btmp</code>, which was over 600MB. Attempting to run</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>watch sudo last -f /var/log/btmp | head
</code></pre></div></div>
<p>showed several attempts per second to brute force into my machine -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root ssh:notty 61.177.172.12 Sun Apr 30 03:21 gone - no logout
root ssh:notty 61.177.172.12 Sun Apr 30 03:20 - 03:21 (00:00)
root ssh:notty 61.177.172.12 Sun Apr 30 03:20 - 03:20 (00:00)
root ssh:notty 61.177.172.12 Sun Apr 30 03:20 - 03:20 (00:00)
root ssh:notty 61.177.172.12 Sun Apr 30 03:20 - 03:20 (00:00)
root ssh:notty 61.177.172.12 Sun Apr 30 03:20 - 03:20 (00:00)
root ssh:notty 218.65.30.38 Sun Apr 30 03:20 - 03:20 (00:00)
root ssh:notty 218.65.30.38 Sun Apr 30 03:20 - 03:20 (00:00)
root ssh:notty 61.177.172.12 Sun Apr 30 03:20 - 03:20 (00:00)
root ssh:notty 218.65.30.38 Sun Apr 30 03:20 - 03:20 (00:00)
</code></pre></div></div>
<p>A search for a solution caused me to conclude that such attempts happen on many public facing server and few solutions exist, such as - using a firewall and allowing only certain hosts login. However, such a thing is not possible to implement on my home server. So, I decided to use port knocking to secure my machine.</p>
<p>Installing a port knocker on ubuntu is as simple as -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt install knockd
</code></pre></div></div>
<p>One would also need <code class="highlighter-rouge">iptables</code> and <code class="highlighter-rouge">iptables-persistent</code>. Next configure iptables as -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo iptables -A INPUT -s 192.168.1.0/24 -p tcp --dport 22 -j ACCEPT
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 22 -j DROP
</code></pre></div></div>
<p>This allows SSH traffic from my local network, viz. 192.168.1.0/24. It further allows traffic from already established connections through. Finally it drops all traffic to SSH that in not already accepted.</p>
<p>We now configure <code class="highlighter-rouge">knockd</code> in the file <code class="highlighter-rouge">/etc/knockd.conf</code></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[options]
LogFile = /var/log/knockd.log
interface = eno1
[openSSH4knocker]
sequence = 64001,64099,64024,64005
seq_timeout = 30
start_command = /sbin/iptables -I INPUT 1 -s %IP% -p tcp --dport 22 -j ACCEPT
tcpflags = syn
cmd_timeout = 180
stop_command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
[openSSH4everyone]
sequence = 64005,64025,64098,64010
seq_timeout = 30
command = /sbin/iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT
tcpflags = syn
cmd_timeout = 180
stop_command = /sbin/iptables -D INPUT -p tcp --dport 22 -j ACCEPT
</code></pre></div></div>
<p>This executes an iptables command to allow traffic through from the IP that knocked on the ports - 64001, 64099, 64024, 64005 in a sequence - openSSH4knocker. On timeout the inserted rule is dropped. This lets the knocking host a window of opportunity to setup a SSH connection and keep it active. The openSSH4everyone opens SSH for all anyone for a period of 180s.</p>
<h2 id="how-to-knock">How to knock?</h2>
<h3 id="windows">Windows</h3>
<p>There are several knocker clients. I chose to use <a href="https://sourceforge.net/projects/knockknock/">KnockKnock</a>. It is not a very good client as it allows on more than one sequence to be stored. I have discovered that I need to keep a second instance of the program in a separate folder to send a second sequence or to a different host. Moreover, I cannot ping a host by name or set the interval between knocks. I absolutely need an IP address. Should I find a better knocker, I would ditch this immediately.</p>
<h3 id="linux">Linux</h3>
<p>Linux has client called knock that is easy to use -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>knock -v host 64001:tcp 64099:tcp 64024:tcp 64005:tcp
</code></pre></div></div>
<p>It would be convenient to store this command with the port sequence in a script.</p>
<h3 id="android">Android</h3>
<p>I am pretty happy with <a href="https://play.google.com/store/apps/details?id=com.xargsgrep.portknocker">Port Knocker</a> on android. It does provide an option to set delays between knocks. This is important as TCP does not guarantee sequence of packet delivery.</p>Bharath Bhushan LohrayRecently I discovered that my log files started growing. The biggest among them was btmp, which was over 600MB. Attempting to runSuspending a computer over SSH2017-04-07T00:00:00-05:002017-04-07T00:00:00-05:00https://ratnavali.net/weblog/suspending-a-computer-over-ssh<p>I wanted to use an automated script to suspend all nodes in my ubuntu cluster. I decided to use the SSH to send a <code class="highlighter-rouge">pm-suspend</code> command to do it. The results were unexpected.</p>
<p>The first time I ran -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>local_user@hostname:~$ ssh remote_user@remote_host sudo pm-suspend
</code></pre></div></div>
<p>I expected to enter the password for <code class="highlighter-rouge">sudo</code>. However, it exited with a message -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo: no tty present and no askpass program specified
</code></pre></div></div>
<p>So, I decided to edit the <code class="highlighter-rouge">sudoers</code> (using <code class="highlighter-rouge">visudo</code>) file to ensure that the <code class="highlighter-rouge">pm-suspend</code> would execute without prompting for the password. I added the follwing line to the <code class="highlighter-rouge">/etc/sudoers</code> file</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>%sudo ALL=(ALL) NOPASSWD:/usr/sbin/pm-suspend
</code></pre></div></div>
<p>After this, I tried the same command and succeed at putting the remote machine to sleep. I could see the power LED of the remote machine blinking in the sleep mode. However, the SSH session did not disconnect. This was undesirable for my purpose. Next I modify my command to background the <code class="highlighter-rouge">pm-suspend</code> process.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>local_user@hostname:~$ ssh remote_user@remote_host "sudo pm-suspend&"
</code></pre></div></div>
<p>The result was no different. The SSH session did not disconnect. The next attempt was to use <code class="highlighter-rouge">nohup</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>local_user@hostname:~$ ssh remote_user@remote_host "nohup sudo pm-suspend"
</code></pre></div></div>
<p>Alas, the SSH session persisted.</p>
<h2 id="the-solution-">The Solution !!!</h2>
<p>Finally, I decided to use the <code class="highlighter-rouge">screen</code> utility with the <code class="highlighter-rouge">-d</code> switch to detach the screen.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>local_user@hostname:~$ ssh remote_user@remote_host "screen -d -m sudo pm-suspend"
</code></pre></div></div>
<p>This time, not only did the remote host enter the sleep state, but the SSH session terminated as desired.</p>Bharath Bhushan LohrayI wanted to use an automated script to suspend all nodes in my ubuntu cluster. I decided to use the SSH to send a pm-suspend command to do it. The results were unexpected.Bootstrap Like Responsive CSS2016-12-12T00:00:00-06:002016-12-12T00:00:00-06:00https://ratnavali.net/weblog/bootstrap-like-responsive-css<p>Several years back, I created my web based CV. It was an awesome website and I was very proud of it. Here are a few screen shots -</p>
<p><img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/education_pg_dt.png" alt="Education - as seen on a desktop" /></p>
<p><strong>Education - as seen on a desktop</strong></p>
<p><img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/projects_pg_dt.png" alt="Project - as seen on a desktop" /></p>
<p><strong>Project - as seen on a desktop</strong></p>
<p><img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/work_pg_dt.png" alt="Work - as seen on a desktop" /></p>
<p><strong>Work - as seen on a desktop</strong></p>
<p>I was very proud of this and told the first person I met to check it out. He whipped out his phone and went to the URL - https://bharath.lohray.com/cv. The page I saw was a huge disappointment!</p>
<p><img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/education_pg_m.png" alt="Education - as seen on a mobile device" /> <img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/projects_pg_m.png" alt="Project - as seen on a mobile device" /> <img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/work_pg_m.png" alt="Work - as seen on a mobile device" /></p>
<p>I had not given a single thought to how the site would look on a mobile device. After all the efforts I had put to get it to work, I had no heart to tweak (rewrite from scratch) the page any further. Now, several years later, I decided to rewrite my online CV this time with the “Mobile First” philosophy in mind. Over the years I discovered that using a pre-built CSS framework was the most predictive way to build a page that would works across all device sizes. I took a special liking to <a href="https://getbootstrap.com/css/">Bootstrap CSS</a>. After a few projects on Bootstrap, I discovered that I was using Bootstrap CSS, jQuery, Bootstrap JS and jQuery UI and add jQuery Mobile to the mix if it were a mobile website.</p>
<p>In spite of all the libraries bloating my page, I still did not get every thing I wanted. I was hesitant to modify any of Bootstrap CSS files as this would be a major hurdle to upgrading Bootstrap with newer versions. So, I decided to take the best of Bootstrap CSS and make it my own. And the best of Bootstrap CSS is it’s responsiveness!</p>
<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">/*
A CSS skeleton extracted from bootstrap.css to define responsive classes.
Make sure you define classes for each screen size and you should have a responsive design :-)
*/</span>
<span class="c">/* print */</span>
<span class="k">@media</span> <span class="n">print</span> <span class="p">{</span>
<span class="p">}</span>
<span class="c">/* xs */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">max-width</span><span class="p">:</span> <span class="m">767px</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
<span class="c">/* sm+ */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">min-width</span><span class="p">:</span> <span class="m">768px</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
<span class="c">/* md+ */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">min-width</span><span class="p">:</span> <span class="m">992px</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
<span class="c">/* lg */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">min-width</span><span class="p">:</span> <span class="m">1200px</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
<span class="c">/* sm */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">min-width</span><span class="p">:</span> <span class="m">768px</span><span class="p">)</span> <span class="n">and</span> <span class="p">(</span><span class="n">max-width</span><span class="p">:</span> <span class="m">991px</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
<span class="c">/* md */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">min-width</span><span class="p">:</span> <span class="m">992px</span><span class="p">)</span> <span class="n">and</span> <span class="p">(</span><span class="n">max-width</span><span class="p">:</span> <span class="m">1199px</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
<span class="c">/* some deprecated IE stuff - don't worry about it! */</span>
<span class="k">@-ms-viewport</span> <span class="p">{</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://gist.github.com/lordloh/88ed45ed8875251e92ba9614300bf180">Here</a> is the gist containing the code.</p>
<p>This CSS skeleton extracted from <code class="highlighter-rouge">bootstrap.css</code> lets me fill in my styles across the various screen sizes and bring responsiveness to my pages with the greatest ease. Here are the results of my 3-2-1 fall back multi-column CSS - very impressive!</p>
<p><img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/projects_pg_lg.png" alt="Large device" /></p>
<p><img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/projects_pg_md.png" alt="Large device" /></p>
<p><img src="https://s3.amazonaws.com/cdn.bharath.lohray.com/weblog/im/bootstrap-like-responsive-css/projects_pg_sm-xs.png" alt="Large device" /></p>Bharath Bhushan LohraySeveral years back, I created my web based CV. It was an awesome website and I was very proud of it. Here are a few screen shots - Education - as seen on a desktop Project - as seen on a desktop Work - as seen on a desktop I was very proud of this and told the first person I met to check it out. He whipped out his phone and went to the URL - https://bharath.lohray.com/cv. The page I saw was a huge disappointment! I had not given a single thought to how the site would look on a mobile device. After all the efforts I had put to get it to work, I had no heart to tweak (rewrite from scratch) the page any further. Now, several years later, I decided to rewrite my online CV this time with the “Mobile First” philosophy in mind. Over the years I discovered that using a pre-built CSS framework was the most predictive way to build a page that would works across all device sizes. I took a special liking to Bootstrap CSS. After a few projects on Bootstrap, I discovered that I was using Bootstrap CSS, jQuery, Bootstrap JS and jQuery UI and add jQuery Mobile to the mix if it were a mobile website. In spite of all the libraries bloating my page, I still did not get every thing I wanted. I was hesitant to modify any of Bootstrap CSS files as this would be a major hurdle to upgrading Bootstrap with newer versions. So, I decided to take the best of Bootstrap CSS and make it my own. And the best of Bootstrap CSS is it’s responsiveness! /* A CSS skeleton extracted from bootstrap.css to define responsive classes. Make sure you define classes for each screen size and you should have a responsive design :-) */ /* print */ @media print { } /* xs */ @media (max-width: 767px) { } /* sm+ */ @media (min-width: 768px) { } /* md+ */ @media (min-width: 992px) { } /* lg */ @media (min-width: 1200px) { } /* sm */ @media (min-width: 768px) and (max-width: 991px) { } /* md */ @media (min-width: 992px) and (max-width: 1199px) { } /* some deprecated IE stuff - don't worry about it! */ @-ms-viewport { } Here is the gist containing the code. This CSS skeleton extracted from bootstrap.css lets me fill in my styles across the various screen sizes and bring responsiveness to my pages with the greatest ease. Here are the results of my 3-2-1 fall back multi-column CSS - very impressive!On Demand Offline Backups2016-10-27T00:00:00-05:002016-10-27T00:00:00-05:00https://ratnavali.net/weblog/on-demand-offline-backups<p>I have a few monthly digital magazine subscriptions that I have been getting for years. I have saved and aggregated on my hard disk. Recently, I lost a chunk of these due to my mistake at the terminal. As a precaution, I had these sync to my cloud storage space on <a href="https://hubic.com/home/new/?referral=UUUSFV">hubic</a> (Offer 25GB of storage space + 50 with 5 referrals). I thought I hand a second copy secured. However, as I logged in to the hubic web interface, I saw files vanishing right before my eyes. The hubic client program syncing between my computer and the cloud was sending instructions to get rid of files that were no longer on the computer. By the time I could log in via SSH and terminate the client program, it was too late. Unlike <a href="https://db.tt/F8PSoGj3">Dropbox</a>, hubic does not keep deleted files to undelete. Years of aggregated magazine subscriptions lost in minutes.</p>
<p>This made me see the necessity to have offline backups.</p>
<p>One easy way was to set <code class="highlighter-rouge">cron</code> tasks to periodically copy certain folders to an external drive via <code class="highlighter-rouge">rsync</code>. However, leaving an external drive connected does turns this “offline” approach into “online”. I felt, it would be ideal if I could plug in my drive into my NAS whenever, I wanted and this would cause a backup job to start running. I looked around and discovered that I could use <a href="https://en.wikipedia.org/wiki/Udev"><code class="highlighter-rouge">udev</code></a>, but reading more, I decided I did not want to dive into <code class="highlighter-rouge">udev</code> for this.</p>
<p>I finally settled on an idea, where I plugin my hard disk into the NAS whenever, I wanted to back up and that night, the selected folders would back up to the hard drive. The next morning, I could unplug the disk and voila, I have my little offline backup system.</p>
<p>I set about implementing this using a shell script that would run every night and check, if my external USB backup, is plugged in or not. If found, it would initiate a backup sequence. Here is the shell script that I used -</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#! /bin/bash</span>
<span class="nv">PART_ID</span><span class="o">=</span><span class="s2">"926fc38e-bc7e-467a-953e-25c6555939a0"</span>
<span class="nv">now</span><span class="o">=</span><span class="k">$(</span><span class="nb">date</span><span class="k">)</span>
<span class="nb">echo</span> <span class="s2">"===== Running at </span><span class="nv">$now</span><span class="s2"> ====="</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-b</span> /dev/disk/by-uuid/<span class="s2">"</span><span class="nv">$PART_ID</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Partition Found!"</span>
<span class="nb">mkdir</span> ~/<span class="nv">$PART_ID</span>
mount <span class="nv">$PART_ID</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> <span class="nt">-eq</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Mounted"</span>
<span class="c"># Run backups</span>
rsync <span class="nt">-r</span> <span class="s2">"/home/bharath/hubic/Datasheets"</span> ~/<span class="nv">$PART_ID</span>/.
rsync <span class="nt">-r</span> <span class="s2">"/home/bharath/hubic/Harvard Business Review"</span> ~/<span class="nv">$PART_ID</span>/.
rsync <span class="nt">-r</span> <span class="s2">"/home/bharath/hubic/IEEE Computer Magazine"</span> ~/<span class="nv">$PART_ID</span>/.
rsync <span class="nt">-r</span> <span class="s2">"/home/bharath/hubic/Linux Journal"</span> ~/<span class="nv">$PART_ID</span>/.
rsync <span class="nt">-r</span> <span class="s2">"/home/bharath/hubic/MagPi"</span> ~/<span class="nv">$PART_ID</span>/.
rsync <span class="nt">-r</span> <span class="s2">"/home/bharath/hubic/Make Magazine"</span> ~/<span class="nv">$PART_ID</span>/.
rsync <span class="nt">-r</span> <span class="s2">"/home/bharath/Photos"</span> ~/<span class="nv">$PART_ID</span>/.
rsync <span class="nt">-rl</span> <span class="s2">"/home/bharath/Datasets"</span> ~/<span class="nv">$PART_ID</span>/.
<span class="nb">sync
</span>umount <span class="nv">$PART_ID</span>
pushb <span class="s2">"Offline backup done."</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> <span class="nt">-eq</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Unmounted"</span>
<span class="nb">rm</span> <span class="nt">-Rf</span> ~/<span class="nv">$PART_ID</span>
<span class="nv">now</span><span class="o">=</span><span class="k">$(</span><span class="nb">date</span><span class="k">)</span>
<span class="nb">echo</span> <span class="s2">"===== Done at </span><span class="nv">$now</span><span class="s2"> ====="</span>
<span class="k">fi
fi
else
</span><span class="nb">echo</span> <span class="s2">":-("</span>
<span class="k">fi</span>
</code></pre></div></div>
<p><a href="https://gist.github.com/lordloh/2305d23f898837e5bac7c9d7dae42cc5">Here</a> is the gist containing the code.</p>
<p>To have this script work, I had to add a line allowing users to mount the disk in the <code class="highlighter-rouge">/etc/fstab</code> file.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>UUID=926fc38e-bc7e-467a-953e-25c6555939a0 /home/bharath/926fe38e-bc7e-467a-959e-25c6535939f0 ext4 noauto,noatime,nodiratime,relatime,user 0 1
</code></pre></div></div>
<p>Apart from this, I had to set the user <code class="highlighter-rouge">bharath</code> (me!) <code class="highlighter-rouge">rwx</code> permissions on the entire disk. A big problem with doing this is that If, I move the disk to a different computer - one that does not have the user <code class="highlighter-rouge">bharath</code> (me!) created with the user id of 1000, someone else, who has <code class="highlighter-rouge">UID=1000</code> would have the <code class="highlighter-rouge">rwx</code> permissions. This is something, I would have to live with till I figure a way around this or write my own portable journaling file system.</p>
<p>To perform a nightly check, I would have to add something like the following line to my crontab -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 4 * * * /home/bharath/do_offline_backup.sh >> offline_backup.log
</code></pre></div></div>
<p>On a side note, comparing hubic to Dropbox, I see hubic having a huge price advantage -</p>
<p>| hubic | Dropbox | AWS S3 (Oregon)|
|———————|—————-|—————-|
| €10/year (100 GB) | $100/year (1TB)| $370/year (1TB)|
| €50/year (10 TB) | | + Data transfer cost |
* 2016 October Prices</p>
<p>And hubic servers are hosted in France, we can reasonably hope that this in out of reach of the various 3 letter government agencies of the US.</p>
<p>The downside of hubic are -</p>
<ol>
<li>Speed - it is slightly slower than dropbox.</li>
<li>UI - not as great a UI as dropbox.</li>
<li>Versioning - no versioning or undelete.</li>
<li>Dropbox paper - no on website editors with hubic.</li>
</ol>
<p>If all you want to do is have an online cloud backup, this is the service for you.</p>Bharath Bhushan LohrayI have a few monthly digital magazine subscriptions that I have been getting for years. I have saved and aggregated on my hard disk. Recently, I lost a chunk of these due to my mistake at the terminal. As a precaution, I had these sync to my cloud storage space on hubic (Offer 25GB of storage space + 50 with 5 referrals). I thought I hand a second copy secured. However, as I logged in to the hubic web interface, I saw files vanishing right before my eyes. The hubic client program syncing between my computer and the cloud was sending instructions to get rid of files that were no longer on the computer. By the time I could log in via SSH and terminate the client program, it was too late. Unlike Dropbox, hubic does not keep deleted files to undelete. Years of aggregated magazine subscriptions lost in minutes. This made me see the necessity to have offline backups. One easy way was to set cron tasks to periodically copy certain folders to an external drive via rsync. However, leaving an external drive connected does turns this “offline” approach into “online”. I felt, it would be ideal if I could plug in my drive into my NAS whenever, I wanted and this would cause a backup job to start running. I looked around and discovered that I could use udev, but reading more, I decided I did not want to dive into udev for this. I finally settled on an idea, where I plugin my hard disk into the NAS whenever, I wanted to back up and that night, the selected folders would back up to the hard drive. The next morning, I could unplug the disk and voila, I have my little offline backup system. I set about implementing this using a shell script that would run every night and check, if my external USB backup, is plugged in or not. If found, it would initiate a backup sequence. Here is the shell script that I used - #! /bin/bash PART_ID="926fc38e-bc7e-467a-953e-25c6555939a0" now=$(date) echo "===== Running at $now =====" if [ -b /dev/disk/by-uuid/"$PART_ID" ]; then echo "Partition Found!" mkdir ~/$PART_ID mount $PART_ID if [ $? -eq 0 ]; then echo "Mounted" # Run backups rsync -r "/home/bharath/hubic/Datasheets" ~/$PART_ID/. rsync -r "/home/bharath/hubic/Harvard Business Review" ~/$PART_ID/. rsync -r "/home/bharath/hubic/IEEE Computer Magazine" ~/$PART_ID/. rsync -r "/home/bharath/hubic/Linux Journal" ~/$PART_ID/. rsync -r "/home/bharath/hubic/MagPi" ~/$PART_ID/. rsync -r "/home/bharath/hubic/Make Magazine" ~/$PART_ID/. rsync -r "/home/bharath/Photos" ~/$PART_ID/. rsync -rl "/home/bharath/Datasets" ~/$PART_ID/. sync umount $PART_ID pushb "Offline backup done." if [ $? -eq 0 ]; then echo "Unmounted" rm -Rf ~/$PART_ID now=$(date) echo "===== Done at $now =====" fi fi else echo ":-(" fi Here is the gist containing the code. To have this script work, I had to add a line allowing users to mount the disk in the /etc/fstab file. UUID=926fc38e-bc7e-467a-953e-25c6555939a0 /home/bharath/926fe38e-bc7e-467a-959e-25c6535939f0 ext4 noauto,noatime,nodiratime,relatime,user 0 1 Apart from this, I had to set the user bharath (me!) rwx permissions on the entire disk. A big problem with doing this is that If, I move the disk to a different computer - one that does not have the user bharath (me!) created with the user id of 1000, someone else, who has UID=1000 would have the rwx permissions. This is something, I would have to live with till I figure a way around this or write my own portable journaling file system. To perform a nightly check, I would have to add something like the following line to my crontab - 0 4 * * * /home/bharath/do_offline_backup.sh >> offline_backup.log On a side note, comparing hubic to Dropbox, I see hubic having a huge price advantage - | hubic | Dropbox | AWS S3 (Oregon)| |———————|—————-|—————-| | €10/year (100 GB) | $100/year (1TB)| $370/year (1TB)| | €50/year (10 TB) | | + Data transfer cost | * 2016 October Prices And hubic servers are hosted in France, we can reasonably hope that this in out of reach of the various 3 letter government agencies of the US. The downside of hubic are - Speed - it is slightly slower than dropbox. UI - not as great a UI as dropbox. Versioning - no versioning or undelete. Dropbox paper - no on website editors with hubic. If all you want to do is have an online cloud backup, this is the service for you.Hardcoding Passwords and git2016-10-21T00:00:00-05:002016-10-21T00:00:00-05:00https://ratnavali.net/weblog/hardcoding-passwords-and-git<p>Recently I was working on building myself a script to download my securities transactions from <a href="http://share.robinhood.com/bharatl">Robinhood</a> and save it as a CSV. I found an <a href="https://github.com/Jamonek/Robinhood">unofficial python library</a> that was reverse engineered from the app. As I started building the script, I realized that I could not host it on git. I had hardcoded my credentials in the code.</p>
<p>My first wish was to be able to add a mark up to the code that git would ignore. Looking around I figured out that git does not yet let users to stop tracking changes to specific lines of the code. Sad. But it would not have been a very effective way as preceding lines may change in future. The only solution is not to hard code usernames and passwords.</p>
<p>The solution I came up with was to store my credentials in a <code class="highlighter-rouge">.json</code> file and to get python read in the <code class="highlighter-rouge">credentials.json</code> file into a dictionary.</p>
<p>Here is my solution -</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'credentials.json'</span><span class="p">)</span> <span class="k">as</span> <span class="n">json_data</span><span class="p">:</span>
<span class="n">credentials</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">json_data</span><span class="p">)</span>
<span class="n">login</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="n">credentials</span><span class="p">[</span><span class="s">'user'</span><span class="p">]</span> <span class="n">password</span><span class="o">=</span><span class="n">credentials</span><span class="p">[</span><span class="s">'password'</span><span class="p">]</span><span class="s">")</span><span class="err">
</span></code></pre></div></div>
<p><a href="https://gist.github.com/lordloh/4a7047049a522570642fd9d941890222">Here is the gist</a> containing the code.</p>
<p><code class="highlighter-rouge">credentials.json</code> is as follows. You may add additional attributed such as API_key etc.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"user"</span><span class="p">:</span><span class="s2">"bharath"</span><span class="p">,</span><span class="w">
</span><span class="s2">"password"</span><span class="p">:</span><span class="s2">"bharath_secret_password"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>I eventually hosted this python script as a CGI on my apache web server. In this case, you can protect your credentials using the following configuration directives -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><Files ~ "credentials.json$">
Require all denied
</Files>
</code></pre></div></div>
<p>Attempting to access <code class="highlighter-rouge">credentials.json</code> now results in a 400 Forbidden error.</p>
<p>Armed with this snippet, I am looking forward to publishing the <em>Hello World</em> of my FinTech application.</p>Bharath Bhushan LohrayRecently I was working on building myself a script to download my securities transactions from Robinhood and save it as a CSV. I found an unofficial python library that was reverse engineered from the app. As I started building the script, I realized that I could not host it on git. I had hardcoded my credentials in the code.Retrieving Historical Stock prices.2016-10-21T00:00:00-05:002016-10-21T00:00:00-05:00https://ratnavali.net/weblog/yql-historical-stock-data<p>Yahoo Open Data Tables is probably the last free source of financial information available after Google shut down it’s finance API a few years back. In a recent project to analyze my stocks trades, I decided to cache 10 year historical prices form the <a href="https://developer.yahoo.com/yql/console/?q=show%20tables&env=store://datatables.org/alltableswithkeys#h=select+*+from+yahoo.finance.historicaldata+where+symbol+%3D+%22YHOO%22+and+startDate+%3D+%222016-10-01%22+and+endDate+%3D+%222016-10-30%22">Yahoo Open Data Tables</a>.</p>
<p>Yahoo API queries are syntactically similar to SQL. For instance, to retrieve the stock prices for “YHOO” for the month of October 2016, I would structure my query as -</p>
<pre><code class="language-SQL">select * from yahoo.finance.historicaldata where symbol = "YHOO" and startDate = "2016-10-01" and endDate = "2016-10-30"
</code></pre>
<p>However, an attempt to set <code class="highlighter-rouge">startDate = "2006-01-01"</code> and <code class="highlighter-rouge">endDate = "2016-10-01"</code> gets me -</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"query"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"count"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="s2">"created"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2016-10-21T05:09:05Z"</span><span class="p">,</span><span class="w">
</span><span class="s2">"lang"</span><span class="p">:</span><span class="w"> </span><span class="s2">"en-US"</span><span class="p">,</span><span class="w">
</span><span class="s2">"results"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>additionally, the diagnostic information reveals that the error thrown is -</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"yahoo.finance.historicaldata"</span><span class="p">,</span><span class="w">
</span><span class="s2">"verb"</span><span class="p">:</span><span class="w"> </span><span class="s2">"select"</span><span class="p">,</span><span class="w">
</span><span class="s2">"content"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Too many instructions executed: 50220753"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>So, the way around was to use map the time range into several 365 day ranges and finally merge the results (reduce). To this end I wore a python function to fetch the data and a wrapper that would perform the map-reduce.</p>
<p>Here is my code -</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">urllib</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span><span class="p">,</span> <span class="n">date</span>
<span class="k">def</span> <span class="nf">yql_stock_history</span><span class="p">(</span><span class="n">symbol</span><span class="p">,</span><span class="n">start_date</span><span class="p">,</span><span class="n">end_date</span><span class="p">):</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="s">"https://query.yahooapis.com/v1/public/yql"</span>
<span class="n">env</span> <span class="o">=</span> <span class="s">"store://datatables.org/alltableswithkeys"</span>
<span class="n">response_format</span> <span class="o">=</span> <span class="s">"json"</span>
<span class="n">query</span><span class="o">=</span><span class="s">'select * from yahoo.finance.historicaldata where symbol = "'</span><span class="o">+</span><span class="n">symbol</span><span class="o">+</span><span class="s">'" and startDate = "'</span><span class="o">+</span><span class="n">start_date</span><span class="o">+</span><span class="s">'" and endDate = "'</span><span class="o">+</span><span class="n">end_date</span><span class="o">+</span><span class="s">'"'</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="s">'format'</span><span class="p">:</span><span class="n">response_format</span><span class="p">,</span> <span class="s">'env'</span><span class="p">:</span><span class="n">env</span><span class="p">,</span> <span class="s">'q'</span><span class="p">:</span><span class="n">query</span><span class="p">}</span>
<span class="n">req_data</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">params</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'ascii'</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">request</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s">'https://query.yahooapis.com/v1/public/yql'</span><span class="p">,</span><span class="n">req_data</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">res</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span>
<span class="n">res</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">except</span> <span class="n">urllib</span><span class="o">.</span><span class="n">error</span><span class="o">.</span><span class="n">URLError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
<span class="n">rise</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">code</span><span class="p">)</span>
<span class="n">json_parsed</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">data</span><span class="p">)[</span><span class="s">'query'</span><span class="p">][</span><span class="s">'results'</span><span class="p">]</span>
<span class="k">if</span> <span class="p">(</span><span class="n">json_parsed</span> <span class="o">!=</span> <span class="bp">None</span><span class="p">):</span>
<span class="k">return</span> <span class="n">json_parsed</span><span class="p">[</span><span class="s">'quote'</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">yql_stock_long_history</span><span class="p">(</span><span class="n">symbol</span><span class="p">,</span> <span class="n">start_date</span><span class="p">,</span> <span class="n">end_date</span><span class="p">):</span>
<span class="n">start_d</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">start_date</span><span class="p">,</span> <span class="s">'</span><span class="si">%</span><span class="s">Y-</span><span class="si">%</span><span class="s">m-</span><span class="si">%</span><span class="s">d'</span><span class="p">)</span>
<span class="n">end_d</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">end_date</span><span class="p">,</span> <span class="s">'</span><span class="si">%</span><span class="s">Y-</span><span class="si">%</span><span class="s">m-</span><span class="si">%</span><span class="s">d'</span><span class="p">)</span>
<span class="n">diff</span><span class="o">=</span><span class="n">end_d</span> <span class="o">-</span> <span class="n">start_d</span>
<span class="n">max_days</span> <span class="o">=</span> <span class="mi">365</span>
<span class="k">if</span> <span class="p">(</span><span class="n">diff</span><span class="o">.</span><span class="n">days</span> <span class="o"><</span> <span class="mi">0</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="k">elif</span><span class="p">(</span><span class="n">diff</span><span class="o">.</span><span class="n">days</span> <span class="o">></span> <span class="n">max_days</span><span class="p">):</span>
<span class="n">data_part</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">start_d</span>
<span class="k">while</span><span class="p">((</span><span class="n">end_d</span><span class="o">-</span><span class="n">s</span><span class="p">)</span><span class="o">.</span><span class="n">days</span> <span class="o">>=</span> <span class="n">max_days</span><span class="p">):</span>
<span class="n">price_history</span><span class="o">=</span><span class="n">yql_stock_history</span><span class="p">(</span><span class="n">symbol</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">s</span><span class="o">+</span><span class="n">timedelta</span><span class="p">(</span><span class="n">max_days</span><span class="p">)))</span>
<span class="n">s</span><span class="o">=</span><span class="n">s</span><span class="o">+</span><span class="n">timedelta</span><span class="p">(</span><span class="n">days</span> <span class="o">=</span> <span class="n">max_days</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="n">price_history</span> <span class="o">==</span> <span class="bp">None</span><span class="p">):</span>
<span class="k">continue</span>
<span class="n">data_part</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">price_history</span><span class="p">)</span>
<span class="n">price_data</span> <span class="o">=</span> <span class="p">[</span><span class="n">b</span> <span class="k">for</span> <span class="n">sublist</span> <span class="ow">in</span> <span class="n">data_part</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">sublist</span><span class="p">]</span>
<span class="n">price_history</span> <span class="o">=</span> <span class="n">yql_stock_history</span><span class="p">(</span><span class="n">symbol</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">s</span><span class="o">+</span><span class="n">timedelta</span><span class="p">(</span><span class="n">max_days</span><span class="p">)))</span>
<span class="n">data_part</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">price_history</span><span class="p">)</span>
<span class="n">price_data</span> <span class="o">=</span> <span class="p">[</span><span class="n">b</span> <span class="k">for</span> <span class="n">sublist</span> <span class="ow">in</span> <span class="n">data_part</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">sublist</span><span class="p">]</span>
<span class="k">return</span> <span class="n">price_data</span>
<span class="k">elif</span><span class="p">(</span><span class="n">diff</span><span class="o">.</span><span class="n">days</span> <span class="o">==</span> <span class="mi">1</span><span class="p">):</span>
<span class="k">return</span> <span class="p">[</span><span class="n">yql_stock_history</span><span class="p">(</span><span class="n">symbol</span><span class="p">,</span> <span class="n">start_date</span><span class="p">,</span> <span class="n">end_date</span><span class="p">)]</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">yql_stock_history</span><span class="p">(</span><span class="n">symbol</span><span class="p">,</span> <span class="n">start_date</span><span class="p">,</span> <span class="n">end_date</span><span class="p">)</span>
</code></pre></div></div>
<p><a href="https://gist.github.com/lordloh/97f2eb5e6e0c6f1291c54b99dae30e8d">Here is the gist</a> containing the code.</p>
<p>The wrapper <code class="highlighter-rouge">yql_stock_long_history</code> is the wrapper that performs the map reduce if the query is too large. Feel free to leave comments on the gist page.</p>Bharath Bhushan LohrayYahoo Open Data Tables is probably the last free source of financial information available after Google shut down it’s finance API a few years back. In a recent project to analyze my stocks trades, I decided to cache 10 year historical prices form the Yahoo Open Data Tables.pip for Cygwin2016-09-25T00:00:00-05:002016-09-25T00:00:00-05:00https://ratnavali.net/weblog/cygwin-pip<p>I have been using cygwin for a while. To say the least, I get to do X11 forwarding on windows. Windows 10 now has an experimental feature called Bash for Windows. I think this was released along with the Windows 10 Anniversary update. However, I have not succeeded in getting X11 to run on it.</p>
<p>This post is about how to install <code class="highlighter-rouge">pip</code> for python on cygwin. Cygwin does not have a package that provides pip as Ubuntu does. But getting pip on Cygwin is really easy. <a href="http://stackoverflow.com/a/31958249/482176">This answer</a> on Stack Overflow outlines the process.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python -m ensurepip
</code></pre></div></div>
<p>Then do the same for python 3</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python3 -m ensurepip
</code></pre></div></div>
<p>Now, upgrade pip to the latest version.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install --upgarede pip
</code></pre></div></div>
<p>The problem with this is that it makes pip2 the default pip and you loose access to pip3. I got around this by -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python3 -m pip install --upgrade pip
</code></pre></div></div>
<p>Phew !!! I can finally install <a href="https://rg3.github.io/youtube-dl/"><code class="highlighter-rouge">youtube-dl</code></a> without hassles -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip3 install youtube-dl
</code></pre></div></div>Bharath Bhushan LohrayI have been using cygwin for a while. To say the least, I get to do X11 forwarding on windows. Windows 10 now has an experimental feature called Bash for Windows. I think this was released along with the Windows 10 Anniversary update. However, I have not succeeded in getting X11 to run on it.Terminal Malfunction2016-09-24T00:00:00-05:002016-09-24T00:00:00-05:00https://ratnavali.net/weblog/terminal-malfunction<p>Occasionally, something I do at the terminal causes the terminal to misbehave at the end of the output. This usually happens when I display a binary file. For example -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat /bin/bash
</code></pre></div></div>
<p>Sometimes, you can reset the terminal from the menu. A few times, this has not worked either. At such times, I usually quit the terminal and start a new window. I did not have to. All I needed to do was -</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>stty sane
</code></pre></div></div>
<p>And the terminal was back to normal behaviour - input echo and all. What does <code class="highlighter-rouge">stty sane</code> do?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sane same as cread -ignbrk brkint -inlcr -igncr icrnl icanon iexten echo echoe echok -echonl -noflsh -ixoff -iutf8 -iuclc -ixany imaxbel -xcase -olcuc -ocrnl opost -ofill onlcr -onocr -onlret nl0 cr0 tab0 bs0 vt0 ff0 isig -tostop -ofdel -echoprt echoctl echoke -extproc -flusho
all special characters to their default values
</code></pre></div></div>
<p>Effectively, this command resets the terminal to all default values. This should save me a lot of trouble in future.</p>Bharath Bhushan LohrayOccasionally, something I do at the terminal causes the terminal to misbehave at the end of the output. This usually happens when I display a binary file. For example -