Tasdik Rahman
2023-10-25T06:43:18+00:00
https://www.tasdikrahman.com
Tasdik Rahman
Oncall in product teams
2023-10-25T00:00:00+00:00
https://www.tasdikrahman.com/2023/10/25/oncall-rotations
<p>I have been oncall for as long as I can remember being in the industry, so far for every organisation I have been part of. Different things have worked at different phases of the organisation and the teams priorities. I thought I would put down some notes over things which I have seen have worked well.</p>
<h1 id="why-do-we-need-to-have-oncall-rotations">Why do we need to have oncall rotations?</h1>
<p>Because simply put, it would help people not burn out of only being oncall unofficially over time, as they get pulled into specifics of systems which they are more aware of.</p>
<p>Having a proper cycle which cycles through all engineers in the team uniformly, will also allow the teams knowledge and toil being distributed on what does the team get paged more often.</p>
<p>While I was at Gojek, I would recommend graduate engineers joining the team(and it was the same that I recommended to the new graduate engineers in my team), to always shadow while doing production support, as it would be a very good avenue to get onboarded to the services which the team would own and have touchpoints for.</p>
<p>Further more, it reduces the bus factor, as knowledge is then forced in a way to be distributed via docs, debugging problems, runbooks over time in the team.</p>
<h1 id="communication-channels">Communication Channels</h1>
<p>To organise comms during an incident, ideally, have channels, where people can update the status of incidents and talk/discuss on what are they doing.</p>
<p>Also is helpful, to have channels, where you run urgent cares for changes like deployment/config changes to help keep track of what changed for folks to have a look in the production systems, in case they have an incident and wanted to see what were the changes that went inside the system.</p>
<h1 id="oncall-is-for-whom">Oncall is for whom</h1>
<p>Apart from the usual pattern of engineers of the product teams who are oncall, potentially it’s also very useful in situations for having senior engineers who have a lot of cross team information to be oncall along with leaders from the teams, as it helps again in distributing the load of oncall.</p>
<h1 id="getting-paged-too-often">Getting paged too often</h1>
<p>This is another post in itself, but reliability and product development pace need not be exclusive from each other. Change in essense might be the simplest way to introduce instability in the system, but that’s the nature of product development and preventing software rot.</p>
<p>Change is inevitable in the system, so the idea is to always make change, easy to make and easy to revert back as much as possible.</p>
<p>If the team is getting paged more often, the ideal path is to focus on preventing that fire first to prevent people from burning out, but balance is key.</p>
<p>More Reliability comes with a cost.</p>
<p>There are also some parts of the system, which a specific person might have more knowledge due to tribal knowledge possibly or simply because they have built it or spent huge amounts of time with it, and it could end up being that the person then gets paged more in-evitably because of that, this is a symptom to fix to distribute the work of oncall load on the person.</p>
<h1 id="primary-and-secondary">Primary and Secondary</h1>
<p>Layering your oncall is helpful in case, there is a miss by the primary to respond to a page, which gets auto escalated to the layer above. If your team can follow the sun pattern for oncall, that’s potentially the best of options to have, to provide coverage for the team.</p>
<h1 id="oncall-as-part-of-sreoperations-teams">Oncall as part of SRE/Operations teams</h1>
<p>There’s not a fixed set of checklists which will work for every team, but on a general note, reducing toil for oncall and having a rotation is a good place to start with and the iterate as we go.</p>
<p>I have also <a href="https://www.tasdikrahman.com/2022/02/21/evolution-of-support-for-infrastructure-teams/">talked about how first responders for teams</a> requests also is ideally good to be spread across the team over a fixed rotation cycle.</p>
<h1 id="ending-notes">Ending notes</h1>
<p>Oncall is a challenging problem but inevitable for teams, as long as you are providing something to the users. Heroism in oncall, along with toil management will be something which you will have to manage in any team nevertheless, as long as you are not completely ignoring these aspects, you are progressing forward with having a better oncall.</p>
<p>There’s a lot of good literature around the internet on how teams should run oncall, and I would highly recommend reading them. As always, I end up learning something new when I read how teams are operating and learn from them.</p>
Keyboard setups over time
2023-09-27T00:00:00+00:00
https://www.tasdikrahman.com/2023/09/27/keyboard-setups-over-time
<p>Over the years, I have used different keyboards for myself, and my setups have changed over time.</p>
<p>This post is just me assimilating over the setups which I have used over the years to reflect back on the memories with each setup.</p>
<h1 id="college-days">College days</h1>
<p>I would use the laptop keyboard itself when I was in college back then, it had a really tiny keyboard and was an ultrabook which didn’t have a CD-ROM.
The setup in my room was also not very ergonomic in the first place as I didn’t have a chair which was ergonomic enough for me with the desk. But it was nevertheless the first setup which I had in all fairness.</p>
<p>For elevating the laptop, from it’s usual level on the table, I would use books to raise it a bit above, but that was also acting up as a helper for the fan for the laptop, which was notorious to get heated up really quickly. One reason being, the fan was just too small and it being that compact, might not have really helped in the whole heatmanagement of the whole machine.</p>
<center><img src="/content/images/2023/09/college_keyboard_setup.jpg" /></center>
<p>This setup was mostly what I used for the rest of college, this image dates back to 2015.</p>
<h1 id="getting-my-first-keyboard">Getting my first keyboard</h1>
<p>Or rather, getting my first set of keyboards, this was the time when I had heard about mechanical keyboards and was excited about having one for myself. In that phase, I got not one but two mechanical keyboards, both of them being Cherry MX Blue.</p>
<p>They did make some noise as compared to my last setup</p>
<p>The two keyboards being</p>
<ul>
<li><a href="https://www.coolermaster.com/catalog/peripheral/keyboards/masterkeys-pro-l/">Coolermaster CoolKeys Pro L</a>, Cherry MX blue switches.</li>
<li>Keycool 84 Mini, Cheery MX blue switches.</li>
</ul>
<p>Here’s a picture of me with both of the keyboards, over at possibly the first mechanical keyboard meetup in Bangalore, dating back to 2019, which also was the start of the <a href="https://reddit.com/r/mkindia">/r/mkindia</a> community.</p>
<center><img src="/content/images/2023/09/blr_keyboard_meetup.jpg" /></center>
<center><img src="/content/images/2023/09/blr_keyboard_meetup_1.jpg" /></center>
<center><img src="/content/images/2023/09/mk_community.jpg" /></center>
<h1 id="change-over-from-cherry-mx-blue-to-cherry-mx-brown-switch">Change over from Cherry MX blue to Cherry MX brown switch</h1>
<p>At some point, I decided to try out the brown switch, as I hadn’t tried it before, it did feel different from the blue switch, but I got used to it over time.</p>
<p>The keyboard ended being a <a href="https://www.mechanical-keyboard.org/cm-storm-quickfire-rapid/">CM storm QuickFire Rapid</a></p>
<center><img src="/content/images/2023/09/cm_storm_setup.jpg" /></center>
<h1 id="current-setup-as-of-september-2023">Current setup as of September 2023</h1>
<p>Had been using my Keycool 84 mini along with my Coolermaster mostly, with a logitech pebble mouse. I really wanted to try the orthogonal layout along with a curved and split setup.</p>
<p>And I ended up with the <a href="https://kinesis-ergo.com/shop/adv360pro/">Kinsesis Advantage 360 Pro</a>, along with the <a href="https://www.logitech.com/de-de/products/mice/mx-vertical-ergonomic-mouse.910-005448.html">logitech MX vertical</a> mouse.</p>
<p>The keyboard certainly has taken me time to get used to for sure, but I have liked it so far, the initial days of using it were super slow, but I am slowly getting back to the touch typing on it.</p>
<center><img src="/content/images/2023/09/kinesis_setup.jpg" /></center>
<p>This is combined with a standing desk and a nice ergonomic chair and a single monitor with a laptop stand. Looking back, the setup has certainly come a long way.</p>
Renewing your root CA with a new root CA such that the older certs signed by old root CA are still valid
2023-05-18T00:00:00+00:00
https://www.tasdikrahman.com/2023/05/18/renewing-root-ca-while-preserving-the-certificates-it-has-signed
<h1 id="context">Context</h1>
<p>If you have a root CA which you used to sign certificates, and if the root certificate is about to expire, the certificates signed by the root CA will also become invalid after the root CA expires even if the certificates signed by it haven’t expired. As every certificate in the chain must remain valid for your certificate to be valid.</p>
<p>Also for example the kube-apiserver when it comes up, it <code class="language-plaintext highlighter-rouge">--client-ca-file</code> while it comes up, where you can pass the root CA.</p>
<h2 id="going-about-this">Going about this</h2>
<p>Now if you end up replacing this cert, you would have to replace all the certs with certs signed by the new root CA in the clients to allow un-interrupted functioning.</p>
<p>The other option is to append the new root CA inside the same file, this will allow the certs signed by both the new root CA and the old root CA to work, allowing time to update the certs for the clients. Why append in the same file? <a href="https://datatracker.ietf.org/doc/html/rfc1422">RFC 1422</a> mentions the same that a pem file can have multiple certificates</p>
<p>This will also allow no changes for any configuration changes being passed for <code class="language-plaintext highlighter-rouge">--client-ca-file</code> for example for the kube-apiserver</p>
<h2 id="generating-the-root-ca-again">Generating the root CA again</h2>
<p>In order for things to work as expected, we would need to generate the root CA with the same serial and v3 extensions, this will allow the certs signed by the older root CA to verify against the root CA</p>
<p>the CSR generated from the old private key and the old root CA</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">CACRT</span><span class="o">=</span>existing-ca.pem
<span class="nv">CAKEY</span><span class="o">=</span>existing-ca-key.pem
<span class="nv">NEWCA</span><span class="o">=</span>renewed-ca.pem
<span class="c"># using the same serial as this is how the client certs signed by the original CA will be respecting the new CA</span>
<span class="nv">serial</span><span class="o">=</span><span class="sb">`</span>openssl x509 <span class="nt">-in</span> <span class="nv">$CACRT</span> <span class="nt">-serial</span> <span class="nt">-noout</span> | <span class="nb">cut</span> <span class="nt">-f2</span> <span class="nt">-d</span><span class="o">=</span><span class="sb">`</span>
<span class="c"># generate the csr from the old private key and the existing root CA</span>
openssl x509 <span class="nt">-x509toreq</span> <span class="nt">-in</span> <span class="nv">$CACRT</span> <span class="nt">-signkey</span> <span class="nv">$CAKEY</span> <span class="nt">-out</span> <span class="nv">$NEWCA</span>.csr
<span class="c"># v3extensions from the last pem file to be used for the new root CA and using the same subject identifier as the original pem file</span>
<span class="nb">cat </span>renewed-ca.pem.conf
<span class="o">[</span> v3_ca <span class="o">]</span>
<span class="nv">basicConstraints</span><span class="o">=</span> critical, CA:TRUE
<span class="nv">keyUsage</span><span class="o">=</span> critical, Certificate Sign, CRL Sign
<span class="nv">subjectKeyIdentifier</span><span class="o">=</span> <the-serial>
openssl x509 <span class="nt">-req</span> <span class="nt">-days</span> 3650 <span class="nt">-in</span> <span class="nv">$NEWCA</span>.csr <span class="nt">-set_serial</span> 0x<span class="nv">$serial</span> <span class="nt">-singkey</span> <span class="nv">$CAKEY</span> <span class="nt">-out</span> <span class="nv">$NEWCA</span>.crt <span class="nt">-extfile</span> ./<span class="nv">$NEWCA</span>.conf <span class="nt">-extensions</span> v3ca
</code></pre></div></div>
<p>And that’s pretty much, the <code class="language-plaintext highlighter-rouge">$NEWCA.crt</code> can be used now</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://stackoverflow.com/questions/9234796/does-the-expiration-status-of-an-issuers-certificate-affect-a-subjects-expirat">https://stackoverflow.com/questions/9234796/does-the-expiration-status-of-an-issuers-certificate-affect-a-subjects-expirat</a></li>
<li><a href="https://stackoverflow.com/questions/40061263/what-is-ca-certificate-and-why-do-we-need-it">https://stackoverflow.com/questions/40061263/what-is-ca-certificate-and-why-do-we-need-it</a></li>
<li><a href="https://security.stackexchange.com/questions/20803/how-does-ssl-tls-work">https://security.stackexchange.com/questions/20803/how-does-ssl-tls-work</a></li>
<li><a href="https://serverfault.com/questions/9708/what-is-a-pem-file-and-how-does-it-differ-from-other-openssl-generated-key-file">https://serverfault.com/questions/9708/what-is-a-pem-file-and-how-does-it-differ-from-other-openssl-generated-key-file</a></li>
</ul>
neovim setup for golang 6 months in
2023-04-29T00:00:00+00:00
https://www.tasdikrahman.com/2023/04/29/nvim-setup-for-golang-6-months-in
<h1 id="so-far">So far</h1>
<p>I have been using my nvim golang setup for around more than 6months now and I am still learning something new everyday while I use it.</p>
<p>coc.vim works really well so far for me. Linting, autocomplete, jumping to definitions back and forth, code folding, checking for references for where a function/method is used, it works out for me well so far.</p>
<h1 id="what-i-am-trying-to-fix-on-my-setup-so-far">What I am trying to fix on my setup so far</h1>
<p>The debugging experience for sure can be improved, I have tried using <a href="https://github.com/leoluz/nvim-dap-go/">nvim-dap-go</a>.</p>
<p>It does work for tests where you would not have a lot of nesting, for example works great for simple <code class="language-plaintext highlighter-rouge">t.Run()</code> calls which are not overly nested (at least from my usage so far) but I have had trouble using it with some <a href="https://github.com/onsi/ginkgo">ginkgo</a> style tests where it’s not able to parse for the parent test when trying to add a debug point, internally it ends up using treesitter to parse the whole file.</p>
<p>Have an open issue here which I added more context in this issue <a href="https://github.com/leoluz/nvim-dap-go/issues/9#issuecomment-1521496733">https://github.com/leoluz/nvim-dap-go/issues/9#issuecomment-1521496733</a>.</p>
<p>My setup for the same is very simple and mostly untouched from the usual config which is out there in the docs</p>
<div class="language-vim highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">lua</span> <span class="p"><<</span>EOF
local dap <span class="p">=</span> require<span class="p">(</span><span class="s1">'dap'</span><span class="p">)</span>
local dapui <span class="p">=</span> require<span class="p">(</span><span class="s2">"dapui"</span><span class="p">)</span>
dapui<span class="p">.</span>setup<span class="p">()</span>
require<span class="p">(</span><span class="s1">'dap-go'</span><span class="p">).</span>setup <span class="p">{</span>
<span class="p">--</span> delve configurations
delve <span class="p">=</span> <span class="p">{</span>
<span class="p">--</span> time <span class="k">to</span> wait <span class="k">for</span> delve <span class="k">to</span> initialize the <span class="k">debug</span> session<span class="p">.</span>
<span class="p">--</span> default <span class="k">to</span> <span class="m">20</span> seconds
initialize_timeout_sec <span class="p">=</span> <span class="m">20</span><span class="p">,</span>
<span class="p">--</span> <span class="k">a</span> <span class="nb">string</span> that defines the port <span class="k">to</span> start delve debugger<span class="p">.</span>
<span class="p">--</span> default <span class="k">to</span> <span class="nb">string</span> <span class="s2">"${port}"</span> which instructs nvim<span class="p">-</span>dap
<span class="p">--</span> <span class="k">to</span> start the process <span class="k">in</span> <span class="k">a</span> random available port
port <span class="p">=</span> <span class="s2">"${port}"</span>
<span class="p">},</span>
dap_configurations <span class="p">=</span> <span class="p">{</span>
<span class="p">{</span>
<span class="nb">type</span> <span class="p">=</span> <span class="s2">"go"</span><span class="p">,</span>
name <span class="p">=</span> <span class="s2">"Attach remote"</span><span class="p">,</span>
<span class="k">mode</span> <span class="p">=</span> <span class="s2">"remote"</span><span class="p">,</span>
request <span class="p">=</span> <span class="s2">"attach"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="p">--</span> You can use nvim<span class="p">-</span>dap events <span class="k">to</span> <span class="k">open</span> <span class="nb">and</span> <span class="k">close</span> the windows automatically
<span class="p">--</span> from<span class="p">:</span> https<span class="p">:</span><span class="sr">//</span>github<span class="p">.</span><span class="k">com</span><span class="sr">/rcarriga/</span>nvim<span class="p">-</span>dap<span class="p">-</span>ui
dap<span class="p">.</span>listeners<span class="p">.</span>after<span class="p">.</span>event_initialized<span class="p">[</span><span class="s2">"dapui_config"</span><span class="p">]</span> <span class="p">=</span> <span class="k">function</span><span class="p">()</span>
dapui<span class="p">.</span><span class="k">open</span><span class="p">()</span>
end
dap<span class="p">.</span>listeners<span class="p">.</span>before<span class="p">.</span>event_terminated<span class="p">[</span><span class="s2">"dapui_config"</span><span class="p">]</span> <span class="p">=</span> <span class="k">function</span><span class="p">()</span>
dapui<span class="p">.</span><span class="k">close</span><span class="p">()</span>
end
dap<span class="p">.</span>listeners<span class="p">.</span>before<span class="p">.</span>event_exited<span class="p">[</span><span class="s2">"dapui_config"</span><span class="p">]</span> <span class="p">=</span> <span class="k">function</span><span class="p">()</span>
dapui<span class="p">.</span><span class="k">close</span><span class="p">()</span>
end
EOF
</code></pre></div></div>
<h1 id="workflow">Workflow</h1>
<p>The flow of usage is pretty out of the box</p>
<ul>
<li>add your breakpoints in the place you want with <code class="language-plaintext highlighter-rouge">:lua require'dap'.toggle_breakpoint()</code> at the places you want</li>
<li>then debug a specific test with <code class="language-plaintext highlighter-rouge">:lua require('dap-go').debug_test()</code></li>
<li>to go over through the breakpoints, you do a <code class="language-plaintext highlighter-rouge">:lua require('dap').continue()</code></li>
<li>and to terminate the process <code class="language-plaintext highlighter-rouge">lua require'dap'.terminate()</code></li>
</ul>
<h1 id="key-maps-being-used">Key maps being used</h1>
<p>I am as of now using the following keymaps</p>
<div class="language-vim highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmap <span class="p"><</span><span class="k">silent</span><span class="p">></span> <span class="p"><</span>leader<span class="p">></span><span class="nb">tb</span> <span class="p">:</span><span class="k">lua</span> require<span class="s1">'dap'</span><span class="p">.</span>toggle_breakpoint<span class="p">()<</span>CR<span class="p">></span>
nmap <span class="p"><</span><span class="k">silent</span><span class="p">></span> <span class="p"><</span>leader<span class="p">></span><span class="k">tc</span> <span class="p">:</span><span class="k">lua</span> require<span class="s1">'dap'</span><span class="p">.</span><span class="k">continue</span><span class="p">()<</span>CR<span class="p">></span>
nmap <span class="p"><</span><span class="k">silent</span><span class="p">></span> <span class="p"><</span>leader<span class="p">></span>tt <span class="p">:</span><span class="k">lua</span> require<span class="s1">'dap'</span><span class="p">.</span>terminate<span class="p">()<</span>CR<span class="p">></span>
nmap <span class="p"><</span><span class="k">silent</span><span class="p">></span> <span class="p"><</span>leader<span class="p">></span>td <span class="p">:</span><span class="k">lua</span> require<span class="p">(</span><span class="s1">'dap-go'</span><span class="p">).</span>debug_test<span class="p">()<</span>CR<span class="p">></span>
</code></pre></div></div>
My vim setup for golang
2022-10-28T00:00:00+00:00
https://www.tasdikrahman.com/2022/10/28/golang-setup-vim
<p>Ok, not vim, but <a href="https://neovim.io/">nvim</a></p>
<h1 id="tldr-what-does-all-this-get-me-in-my-setup">tl;dr what does all this get me in my setup</h1>
<p><a href="https://github.com/tasdikrahman/dotfiles/tree/master/vim">https://github.com/tasdikrahman/dotfiles/tree/master/vim</a></p>
<ul>
<li>jump to definitions</li>
<li>jump to references</li>
<li>jump to symbols</li>
<li>fuzzy file search</li>
<li>code folding</li>
<li>jumping between test and implementation file</li>
<li>testing specific function</li>
<li>real time code linting</li>
</ul>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Cleaning my vim config after some time and ended up removing a bunch of things and starting afresh, a couple of plugins which had been archived but worked all the while were <a href="https://t.co/lmjXtN7HbS">https://t.co/lmjXtN7HbS</a>, switched to <a href="https://t.co/7ZSoKMgwBW">https://t.co/7ZSoKMgwBW</a> as it was the recommended replacement (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1585959045122621440?ref_src=twsrc%5Etfw">October 28, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Cleaned up my <a href="https://www.vim.org/">vim</a> configuration which I was trying to set up for golang specifically. I did have <a href="https://github.com/fatih/vim-go">vim-go</a> already setup, but wanted to try out <a href="https://github.com/neoclide/coc.nvim">coc.vim</a> along with <a href="https://github.com/golang/tools/tree/master/gopls">gopls</a> since it was around for sometime now. There were also a bunch of plugins which had been archived like <a href="https://github.com/vim-syntastic/syntastic">syntastic</a></p>
<p>This setup will evolve and has space for a couple of things which I will incrementally add over time in the next few weeks.</p>
<h1 id="impressions-so-far">Impressions so far</h1>
<p>I turned off the lsp settings for ale, so that vim.coc would take it over, haven’t removed vim-go completely as I still use a couple of things from there.</p>
<p>The jump of definitions after moving over to gopls, and vim.coc giving over jumping to references, definitions, godocs comes almost the same to the setup for an IDE, along with being able to fold along with using <a href="https://github.com/nvim-treesitter/nvim-treesitter">nvim-treesitter</a> is something which adds up really well along with <a href="https://github.com/kien/ctrlp.vim">ctrl+p</a></p>
<h1 id="changes-and-things-kept-from-existing-vim-go-setup">Changes and things kept from existing vim-go setup</h1>
<p>Turned off the go autoimports functionality which is given over by vim-coc</p>
<p>The go-build and go-run keybindings which I had are already being used, same for the :GoAlternate keybindings to jump and split the panes when going over test and implementation files, along with the highlight configs is something which I have kept.</p>
<p>I did end up removing ultisnips which is one of the plugins presented in vim-go, with coc-go and coc-snippets, as they almost provide the same thing.</p>
<h1 id="what-does-it-lack-as-of-now">What does it lack as of now</h1>
<p>I want to setup up delve as that’s what I feel is remaining as of now for a full out IDE setup which is what I was anyways using, this remains as of now. Will see how <a href="https://github.com/mfussenegger/nvim-dap">nvim-dap</a> fixes up with my current setup and update this section with what I ended up with.</p>
<h1 id="links">Links</h1>
<p>The latest vim config being used by me and what I did to set it up can be found here in the following link.</p>
<ul>
<li><a href="https://github.com/tasdikrahman/dotfiles/tree/master/vim">https://github.com/tasdikrahman/dotfiles/tree/master/vim</a></li>
</ul>
Working remotely in a geographically distributed team without burning yourself out
2022-10-08T00:00:00+00:00
https://www.tasdikrahman.com/2022/10/08/remote-working-in-geographically-distributed-team-without-burning-yourself-out
<h1 id="context">Context</h1>
<p>Wanted to pick ideas from folks who have worked in such setups effectively/led teams over the years across geographical continents. Given I recently took up a fully remote role, with the team that I am joining being spread across EU and in the US as of now.</p>
<p>This post is just for documentary purposes for me to look back how I have fared over the course of this year and the next as I take this journey of working in a remote first company.</p>
<h2 id="takeaways-from-my-discussion-with-a-couple-of-folks">Takeaways from my discussion with a couple of folks.</h2>
<ul>
<li>different machines for work and personal work</li>
<li>respecting your own boundaries of fixed time slots/work times which you work in the day.</li>
<li>having 1-1’s with folks in your team and your other team members who work with you from other teams.</li>
<li>written first communication</li>
<li>meet personally in the office helps further while working remotely.</li>
<li>talking with your respective manager to get a sense of how they want to drive the working style of the team and how they want you to work.</li>
<li>see what are you trying to balance along with your work times and see how they align and make sure you communicate it.</li>
<li>1-1 with your manager is you time since this is for the reportee to convey concerns/ideas to the person you are reporting to</li>
<li>not checking work devices frequently outside of work hours.</li>
<li>understand the set of practices already being followed by your team, if it has already been working remotely for sometime and follow them</li>
</ul>
Scaling cluster upgrades for kubernetes
2022-09-26T00:00:00+00:00
https://www.tasdikrahman.com/2022/09/26/scaling-cluster-upgrades-for-kubernetes
<p>This post is more of a continuation of the talk I gave over at <a href="https://www.meetup.com/kubernetes-openshift-india-meetup/events/288277755/">kubernetes bangalore k8s september 2022 meetup</a>.</p>
<p>Here are the slides, which you can take a peek over, to complement this post, if you would like to go through it before reading further.</p>
<script async="" class="speakerdeck-embed" data-id="5c166502e7e0418dad72eb6c65849ca0" data-ratio="1.77725118483412" src="//speakerdeck.com/assets/embed.js"></script>
<h2 id="context">Context</h2>
<p>I will not repeat the content which is already there in the slides. Will also update this post with the talk link when the talk gets uploaded. But I do want to delve over into the idea of how I feel I would attempt to structure the upgrades next. This post is more on the infrastructure upgrade complexities arising from when managing double digit or more k8s clusters.</p>
<h3 id="what-is-the-bottleneck-at-the-end-of-the-day">What is the bottleneck at the end of the day</h3>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">To just add to the last idea of the main tweet. Having seen and maintained the terraform setup to create/maintain clusters at a couple of places, and this way of doing it doesn't work very well after you start reaching cluster nos in 10's or 100's and more (1/n) <a href="https://t.co/C9cSaE812Q">https://t.co/C9cSaE812Q</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1521420689206648832?ref_src=twsrc%5Etfw">May 3, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>We have a state for the cluster, which also ends up being managed by the way we add/delete/create the cluster with, this could be terraform or eksctl config or cloudformation, whatever you have chosen.</p>
<p>The way to introduce change in these types of setups is often more than not a human interaction happening over with them, it could be a terraform plan, apply for example in this case and the state of the change then gets stored in the state file.</p>
<p>Now imagine having to do the same over 10s or 100s of cluster.</p>
<p>I am increasingly leaning towards a way to manage/create the state of the k8s cluster, via some form of a control loop, whereas the whole terraform style of state is to have interrupts around it. While it can very well be engineered to reduce the interrupts, fundamentally I feel the difference in the way to operate the state is different from the terraform style and control loop style management.</p>
<p>This tweet further discusses this idea.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">In general I prefer control-loop style Management of infrastructure, especially at scale. It’s easier to manage drift and be proactive about modifications. One-shot stuff like terraform gets fragile over time and it’s harder to evolve, turns out software is OK too.</p>— Lincoln Stoll (@lstoll) <a href="https://twitter.com/lstoll/status/1521412277152460800?ref_src=twsrc%5Etfw">May 3, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h3 id="ways-to-think-about-efficiently-managing-this-drift-and-human-bottleneck-problem">Ways to think about efficiently managing this drift and human bottleneck problem</h3>
<p>As described in the last tweet, the control loop managing the state of the infrastructure in question, makes us think more in the direction in which [cluster-api] tries thinking around managing infrastructure. I have come across <a href="https://github.com/openshift/managed-upgrade-operator">https://github.com/openshift/managed-upgrade-operator</a>, which is the closest I have seen an OSS implementation going towards a complete implementation of this style of operation. Reasonably well documented and how I would probably lean into solutioning this problem for a large scale cluster sprawl when trying to solve it the next time.</p>
<p>While there are more examples of automation of rolling upgrades for node groups out there in the form of <a href="https://github.com/keikoproj/upgrade-manager">https://github.com/keikoproj/upgrade-manager</a>, taking over the control loop approach for a specific subset of the problem statement of the upgrade process, in this case the node operations.</p>
<p>Along with <a href="https://github.com/hellofresh/eks-rolling-update">https://github.com/hellofresh/eks-rolling-update</a> and <a href="https://github.com/deliveryhero/k8s-cluster-upgrade-tool">https://github.com/deliveryhero/k8s-cluster-upgrade-tool</a>(P.S. is written by me), both of which are human operated, while although do solve the problem in case, a subset of it if not all, but the openshift’s implementation is a more human interrupt free way to go about the cluster upgrade process, which is what will allow the cluster upgrade process to be sustainably executed at the end of the day, without a lot of toil and effort. Even though if we end up adding the operations of cluster upgrade process further to this way, operations like control plane upgrade, upgrade of the managed node groups and k8s components, we have to at the end of the day, wrangle with the state and modifying it further with the tools which we would have created/managing the specific k8s resources.</p>
<h3 id="closing-thoughts">Closing thoughts</h3>
<p>The control loop strategy would make us the question or reconsider how we are creating and managing these k8s specific resources like control plane for the managed provider/self hosted cluster, managed node groups/self hosted node groups and the k8s components managed by helm. In any case, I foresee someone wanting to consolidate their automation or creation/management process back to the control loop automation which they have in order to have standardisation over time. More of this automation introduced would more more moving codepieces leading to maintenance creep, but the cost of which would be justifiable if you are measurably reducing the effort of the cluster upgrade process for the x number of clusters you own/manage/run.</p>
<p>The whole IAC being run via atlantis and having terragrunt to run plan/apply in the CI is a not so uncommon of a setup as of now, but the churn of cluster upgrades and the complexity which comes up with it, is the not the same as managing a couple of managed services via the terraform config, upgrade/delete of which is most of the times a one step plan/apply.</p>
<p>While to each to their own setup. But in general, reaching to a state where k8s clusters itself are not pets, is a fairly complex problem. If you have already reached that state, then your state of a new cluster creation is very mature, cleaning up of which might be the next logical step to think about, after which you can do a blue-green deployment of the new cluster setup. But again, I have not seen this work out successfully so far, hence, don’t think of your k8s cluster as a pod by this extension of thinking. Save the state of the cluster in a way in which you can modify it over time with your tooling.</p>
<p>If you have implemented some other setup for a large sprawl of k8s clusters for yourself. Would love to hear more about it.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://github.com/openshift/managed-upgrade-operator/blob/master/docs/design.md">https://github.com/openshift/managed-upgrade-operator/blob/master/docs/design.md</a></li>
<li><a href="https://github.com/openshift/managed-upgrade-operator">https://github.com/openshift/managed-upgrade-operator</a></li>
<li><a href="https://github.com/hellofresh/eks-rolling-update">https://github.com/hellofresh/eks-rolling-update</a></li>
<li><a href="https://github.com/deliveryhero/k8s-cluster-upgrade-tool">https://github.com/deliveryhero/k8s-cluster-upgrade-tool</a></li>
</ul>
Musings with client-go of k8s
2022-09-22T00:00:00+00:00
https://www.tasdikrahman.com/2022/09/22/musings-with-client-go-kubernetes
<p>This post mostly is for documentary purposes for myself, about a few things which I ended up noticing while using <a href="https://github.com/kubernetes/client-go">client-go</a> as I used it for <a href="https://github.com/deliveryhero/k8s-cluster-upgrade-tool">deliveryhero/k8s-cluster-upgrade-tool</a>, which used the out-cluster client configuration, a couple of things are specific to that setup, like client init, but other things like testing interactions via client-go are more generic.</p>
<h2 id="initialization-of-the-config">Initialization of the config</h2>
<p>client-go in itself, shows a couple of example of client init <a href="https://github.com/kubernetes/client-go/blob/release-1.21/examples/out-of-cluster-client-configuration/main.go#L44-L62">here</a>, pasting the snippet here for context</p>
<div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">...</span>
<span class="k">var</span> <span class="n">kubeconfig</span> <span class="o">*</span><span class="kt">string</span>
<span class="k">if</span> <span class="n">home</span> <span class="o">:=</span> <span class="n">homedir</span><span class="o">.</span><span class="n">HomeDir</span><span class="p">();</span> <span class="n">home</span> <span class="o">!=</span> <span class="s">""</span> <span class="p">{</span>
<span class="n">kubeconfig</span> <span class="o">=</span> <span class="n">flag</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"kubeconfig"</span><span class="p">,</span> <span class="n">filepath</span><span class="o">.</span><span class="n">Join</span><span class="p">(</span><span class="n">home</span><span class="p">,</span> <span class="s">".kube"</span><span class="p">,</span> <span class="s">"config"</span><span class="p">),</span> <span class="s">"(optional) absolute path to the kubeconfig file"</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">kubeconfig</span> <span class="o">=</span> <span class="n">flag</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"kubeconfig"</span><span class="p">,</span> <span class="s">""</span><span class="p">,</span> <span class="s">"absolute path to the kubeconfig file"</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">flag</span><span class="o">.</span><span class="n">Parse</span><span class="p">()</span>
<span class="c">// use the current context in kubeconfig</span>
<span class="n">config</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">clientcmd</span><span class="o">.</span><span class="n">BuildConfigFromFlags</span><span class="p">(</span><span class="s">""</span><span class="p">,</span> <span class="o">*</span><span class="n">kubeconfig</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="nb">panic</span><span class="p">(</span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">())</span>
<span class="p">}</span>
<span class="c">// create the clientset</span>
<span class="n">clientset</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">kubernetes</span><span class="o">.</span><span class="n">NewForConfig</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="nb">panic</span><span class="p">(</span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">())</span>
<span class="p">}</span>
<span class="o">...</span>
</code></pre></div></div>
<p>This would by default go ahead and initialize the client with the current k8s context you are attached to. For example in your <code class="language-plaintext highlighter-rouge">~/.kube/config</code> file</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
current-context: foo-cluster
kind: Config
preferences: <span class="o">{}</span>
...
</code></pre></div></div>
<p>with the above, the client will get initliazed with the k8s context being <code class="language-plaintext highlighter-rouge">foo-cluster</code>.</p>
<h3 id="initiliazation-happening-by-default-to-the-default-kube-context">Initiliazation happening by default to the default kube context</h3>
<p>What ends up happening underneath is that, the method <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L613">BuildConfigFromFlags()</a>, which goes on to call <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L134">ClientConfig()</a>, which is where the k8s context is deduced, when the call to <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L144">getContext()</a> gets made. The flow defaults to the current context via the call made to <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L462">getContextName()</a>.</p>
<p>This is also where we notice that <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L427">we can add an override</a>, which if added, would select that particular context, instead of choosing the default, which is picked up from the current context already selected in the <code class="language-plaintext highlighter-rouge">~/.kube/context</code>. The only thing is this override is not accessible directly from the BuildConfigFromFlags method, which is what we were using initially, checking further, I didn’t find anything which would help with this in the package, or maybe I missed something feel free to point it out.</p>
<p>But how do we go about overriding this?</p>
<h3 id="initializing-client-go-to-a-user-specified-k8s-context">Initializing client-go to a user specified k8s context</h3>
<p>We would just need to have the <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/overrides.go#L35">Context</a> added to <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/overrides.go#L30">ConfigOverrides</a>, when the call to ClientConfig() gets made at the in <a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L613">BuildConfigFromFlags()</a></p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// buildConfigFromFlags returns the config using which the client will be initialized with the k8s context we want to use</span>
<span class="k">func</span> <span class="n">buildConfigFromFlags</span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">kubeconfigPath</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="n">rest</span><span class="o">.</span><span class="n">Config</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">clientcmd</span><span class="o">.</span><span class="n">NewNonInteractiveDeferredLoadingClientConfig</span><span class="p">(</span>
<span class="o">&</span><span class="n">clientcmd</span><span class="o">.</span><span class="n">ClientConfigLoadingRules</span><span class="p">{</span><span class="n">ExplicitPath</span><span class="o">:</span> <span class="n">kubeconfigPath</span><span class="p">},</span>
<span class="o">&</span><span class="n">clientcmd</span><span class="o">.</span><span class="n">ConfigOverrides</span><span class="p">{</span>
<span class="n">CurrentContext</span><span class="o">:</span> <span class="n">context</span><span class="p">,</span>
<span class="p">})</span><span class="o">.</span><span class="n">ClientConfig</span><span class="p">()</span>
<span class="p">}</span>
<span class="c">// KubeClientInit returns back clientSet</span>
<span class="k">func</span> <span class="n">KubeClientInit</span><span class="p">(</span><span class="n">kubeContext</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="n">kubernetes</span><span class="o">.</span><span class="n">Clientset</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">var</span> <span class="n">kubeConfig</span> <span class="o">*</span><span class="kt">string</span>
<span class="k">if</span> <span class="n">home</span> <span class="o">:=</span> <span class="n">homedir</span><span class="o">.</span><span class="n">HomeDir</span><span class="p">();</span> <span class="n">home</span> <span class="o">!=</span> <span class="s">""</span> <span class="p">{</span>
<span class="n">kubeConfig</span> <span class="o">=</span> <span class="n">flag</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"kubeconfig"</span><span class="p">,</span> <span class="n">filepath</span><span class="o">.</span><span class="n">Join</span><span class="p">(</span><span class="n">home</span><span class="p">,</span> <span class="s">".kube"</span><span class="p">,</span> <span class="s">"config"</span><span class="p">),</span> <span class="s">"(optional) absolute path to the kubeconfig file"</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">kubeConfig</span> <span class="o">=</span> <span class="n">flag</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"kubeconfig"</span><span class="p">,</span> <span class="s">""</span><span class="p">,</span> <span class="s">"absolute path to the kubeconfig file"</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">flag</span><span class="o">.</span><span class="n">Parse</span><span class="p">()</span>
<span class="n">config</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">buildConfigFromFlags</span><span class="p">(</span><span class="n">kubeContext</span><span class="p">,</span> <span class="o">*</span><span class="n">kubeConfig</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="o">&</span><span class="n">kubernetes</span><span class="o">.</span><span class="n">Clientset</span><span class="p">{},</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="s">"error building the config for building the client-set for client-go"</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">// create the clientset</span>
<span class="n">clientSet</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">kubernetes</span><span class="o">.</span><span class="n">NewForConfig</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="o">&</span><span class="n">kubernetes</span><span class="o">.</span><span class="n">Clientset</span><span class="p">{},</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="s">"error building the client-set for client-go"</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">clientSet</span><span class="p">,</span> <span class="no">nil</span>
<span class="p">}</span>
</code></pre></div></div>
<p>the init would then look like</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">...</span>
<span class="n">kubeClient</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">k8s</span><span class="o">.</span><span class="n">KubeClientInit</span><span class="p">(</span><span class="s">"cluster-name"</span><span class="p">)</span>
<span class="o">...</span>
</code></pre></div></div>
<h2 id="testing-client-go-interactions">Testing client-go interactions</h2>
<p>While writing a couple of interactions on <a href="https://github.com/deliveryhero/k8s-cluster-upgrade-tool">deliveryhero/k8s-cluster-upgrade-tool</a>, ended up checking what were folks doing to add specs for interactions with client-go, and we already had the <a href="https://pkg.go.dev/k8s.io/client-go/kubernetes/fake">https://pkg.go.dev/k8s.io/client-go/kubernetes/fake</a> package out there which one could use to test the interactions with the client.</p>
<p>My specific cases for testing were quite simple to setup. The first thing again, which you would need to take care of is obviously, that the method which you want to test, should be able to take the client as a dependency which you inject in the spec. After that’s it’s super simple.</p>
<p>For example, for one case, I would want a behaviour, where querying a specific object type, when queried with specific attributes, would return me back that object. Then the methods behaviour of how it processed that output would be put under test.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">...</span>
<span class="n">client</span> <span class="o">:=</span> <span class="n">fake</span><span class="o">.</span><span class="n">NewSimpleClientset</span><span class="p">(</span><span class="o">&</span><span class="n">appsv1</span><span class="o">.</span><span class="n">DaemonSet</span><span class="p">{</span>
<span class="n">ObjectMeta</span><span class="o">:</span> <span class="n">metav1</span><span class="o">.</span><span class="n">ObjectMeta</span><span class="p">{</span>
<span class="n">Name</span><span class="o">:</span> <span class="s">"aws-node"</span><span class="p">,</span>
<span class="n">Namespace</span><span class="o">:</span> <span class="s">"kube-system"</span><span class="p">,</span>
<span class="p">},</span>
<span class="n">Spec</span><span class="o">:</span> <span class="n">appsv1</span><span class="o">.</span><span class="n">DaemonSetSpec</span><span class="p">{</span>
<span class="n">Template</span><span class="o">:</span> <span class="n">corev1</span><span class="o">.</span><span class="n">PodTemplateSpec</span><span class="p">{</span>
<span class="n">Spec</span><span class="o">:</span> <span class="n">corev1</span><span class="o">.</span><span class="n">PodSpec</span><span class="p">{</span>
<span class="n">Containers</span><span class="o">:</span> <span class="p">[]</span><span class="n">corev1</span><span class="o">.</span><span class="n">Container</span><span class="p">{</span>
<span class="p">{</span>
<span class="n">Image</span><span class="o">:</span> <span class="s">"aws-node:v1.0.0"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">})</span>
<span class="o">...</span>
</code></pre></div></div>
<p>Now any interactions which we would want to have with the client, where if we query for this object via the specific api’s we would get this object back.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">myMethod</span><span class="p">(</span><span class="n">k8sClient</span> <span class="n">kubernetes</span><span class="o">.</span><span class="n">Interface</span><span class="p">,</span> <span class="n">myObjects</span><span class="o">...</span><span class="p">)</span> <span class="p">{</span>
<span class="c">// when the subject is under test, this method when passed the fake client with the above initialization would have the aws-node object present</span>
<span class="c">// which would be returned.</span>
<span class="n">daemonSet</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">k8sClient</span><span class="o">.</span><span class="n">AppsV1</span><span class="p">()</span><span class="o">.</span><span class="n">DaemonSets</span><span class="p">(</span><span class="n">namespace</span><span class="p">)</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">TODO</span><span class="p">(),</span> <span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">metav1</span><span class="o">.</span><span class="n">GetOptions</span><span class="p">{})</span>
<span class="o">...</span>
<span class="o">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>To give a more full example, here is a case where function returns the container image of the first container for a deployment of a daemonset object</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">GetContainerImageForK8sObject</span><span class="p">(</span><span class="n">k8sClient</span> <span class="n">kubernetes</span><span class="o">.</span><span class="n">Interface</span><span class="p">,</span> <span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">k8sObject</span><span class="p">,</span> <span class="n">namespace</span> <span class="kt">string</span><span class="p">)</span> <span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="n">k8sObject</span> <span class="p">{</span>
<span class="k">case</span> <span class="s">"deployment"</span><span class="o">:</span>
<span class="c">// NOTE: Not targeting other api versions for the objects as of now.</span>
<span class="n">deployment</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">k8sClient</span><span class="o">.</span><span class="n">AppsV1</span><span class="p">()</span><span class="o">.</span><span class="n">Deployments</span><span class="p">(</span><span class="n">namespace</span><span class="p">)</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">TODO</span><span class="p">(),</span> <span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">metav1</span><span class="o">.</span><span class="n">GetOptions</span><span class="p">{})</span>
<span class="k">if</span> <span class="n">k8sErrors</span><span class="o">.</span><span class="n">IsNotFound</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"Deployment %s in namespace %s not found</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">namespace</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="n">statusError</span><span class="p">,</span> <span class="n">isStatus</span> <span class="o">:=</span> <span class="n">err</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">k8sErrors</span><span class="o">.</span><span class="n">StatusError</span><span class="p">);</span> <span class="n">isStatus</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"Error getting deployment %s in namespace %s: %v</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
<span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">namespace</span><span class="p">,</span> <span class="n">statusError</span><span class="o">.</span><span class="n">ErrStatus</span><span class="o">.</span><span class="n">Message</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"there was an error while retrieving the container image"</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">// NOTE: This assumes there is only one container in the k8s object, which is true for the components for us at moment</span>
<span class="k">return</span> <span class="n">deployment</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Template</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Containers</span><span class="p">[</span><span class="m">0</span><span class="p">]</span><span class="o">.</span><span class="n">Image</span><span class="p">,</span> <span class="no">nil</span>
<span class="k">case</span> <span class="s">"daemonset"</span><span class="o">:</span>
<span class="c">// NOTE: Not targeting other api versions for the objects as of now.</span>
<span class="n">daemonSet</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">k8sClient</span><span class="o">.</span><span class="n">AppsV1</span><span class="p">()</span><span class="o">.</span><span class="n">DaemonSets</span><span class="p">(</span><span class="n">namespace</span><span class="p">)</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">TODO</span><span class="p">(),</span> <span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">metav1</span><span class="o">.</span><span class="n">GetOptions</span><span class="p">{})</span>
<span class="k">if</span> <span class="n">k8sErrors</span><span class="o">.</span><span class="n">IsNotFound</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"daemonset %s in namespace %s not found</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">namespace</span><span class="p">)</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="n">statusError</span><span class="p">,</span> <span class="n">isStatus</span> <span class="o">:=</span> <span class="n">err</span><span class="o">.</span><span class="p">(</span><span class="o">*</span><span class="n">k8sErrors</span><span class="o">.</span><span class="n">StatusError</span><span class="p">);</span> <span class="n">isStatus</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"Error getting daemonset %s in namespace %s: %v</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
<span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">namespace</span><span class="p">,</span> <span class="n">statusError</span><span class="o">.</span><span class="n">ErrStatus</span><span class="o">.</span><span class="n">Message</span><span class="p">))</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"there was an error while retrieving the container image"</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">// NOTE: This assumes there is only one container in the k8s object, which is true for the components for us at moment</span>
<span class="k">return</span> <span class="n">daemonSet</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Template</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Containers</span><span class="p">[</span><span class="m">0</span><span class="p">]</span><span class="o">.</span><span class="n">Image</span><span class="p">,</span> <span class="no">nil</span>
<span class="k">default</span><span class="o">:</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"please choose between Daemonset or Deployment k8sobject as they are currently supported"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Specific case when the object is of kind <code class="language-plaintext highlighter-rouge">Deployment</code></p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">TestGetContainerImageForK8sObjectWhenK8sObjectIsDeployment</span><span class="p">(</span><span class="n">t</span> <span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span> <span class="p">{</span>
<span class="k">type</span> <span class="n">deploymentArgs</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">k8sObject</span> <span class="kt">string</span>
<span class="n">k8sObjectName</span> <span class="kt">string</span>
<span class="n">kubeContext</span> <span class="kt">string</span>
<span class="n">namespace</span> <span class="kt">string</span>
<span class="n">deployment</span> <span class="o">*</span><span class="n">appsv1</span><span class="o">.</span><span class="n">Deployment</span>
<span class="p">}</span>
<span class="n">tests</span> <span class="o">:=</span> <span class="p">[]</span><span class="k">struct</span> <span class="p">{</span>
<span class="n">name</span> <span class="kt">string</span>
<span class="n">args</span> <span class="n">deploymentArgs</span>
<span class="n">err</span> <span class="kt">error</span>
<span class="n">output</span> <span class="kt">string</span>
<span class="p">}{</span>
<span class="p">{</span>
<span class="n">name</span><span class="o">:</span> <span class="s">"When the object is of type deployment, the objectname is cluster-autoscaler, object exists and returns back the image"</span><span class="p">,</span>
<span class="n">args</span><span class="o">:</span> <span class="n">deploymentArgs</span><span class="p">{</span><span class="n">k8sObject</span><span class="o">:</span> <span class="s">"deployment"</span><span class="p">,</span> <span class="n">k8sObjectName</span><span class="o">:</span> <span class="s">"cluster-autoscaler"</span><span class="p">,</span> <span class="n">kubeContext</span><span class="o">:</span> <span class="s">"test-context"</span><span class="p">,</span> <span class="n">namespace</span><span class="o">:</span> <span class="s">"kube-system"</span><span class="p">,</span>
<span class="n">deployment</span><span class="o">:</span> <span class="o">&</span><span class="n">appsv1</span><span class="o">.</span><span class="n">Deployment</span><span class="p">{</span>
<span class="n">ObjectMeta</span><span class="o">:</span> <span class="n">metav1</span><span class="o">.</span><span class="n">ObjectMeta</span><span class="p">{</span>
<span class="n">Name</span><span class="o">:</span> <span class="s">"cluster-autoscaler"</span><span class="p">,</span>
<span class="n">Namespace</span><span class="o">:</span> <span class="s">"kube-system"</span><span class="p">,</span>
<span class="p">},</span>
<span class="n">Spec</span><span class="o">:</span> <span class="n">appsv1</span><span class="o">.</span><span class="n">DeploymentSpec</span><span class="p">{</span>
<span class="n">Template</span><span class="o">:</span> <span class="n">corev1</span><span class="o">.</span><span class="n">PodTemplateSpec</span><span class="p">{</span>
<span class="n">Spec</span><span class="o">:</span> <span class="n">corev1</span><span class="o">.</span><span class="n">PodSpec</span><span class="p">{</span>
<span class="n">Containers</span><span class="o">:</span> <span class="p">[]</span><span class="n">corev1</span><span class="o">.</span><span class="n">Container</span><span class="p">{</span>
<span class="p">{</span>
<span class="n">Image</span><span class="o">:</span> <span class="s">"cluster-autoscaler:v1.0.0"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">}},</span>
<span class="n">output</span><span class="o">:</span> <span class="s">"cluster-autoscaler:v1.0.0"</span><span class="p">,</span>
<span class="n">err</span><span class="o">:</span> <span class="no">nil</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="n">name</span><span class="o">:</span> <span class="s">"When the object is of type deployment, the objectname is cluster-autoscaler, object doesn't exist, returns back error"</span><span class="p">,</span>
<span class="n">args</span><span class="o">:</span> <span class="n">deploymentArgs</span><span class="p">{</span><span class="n">k8sObject</span><span class="o">:</span> <span class="s">"deployment"</span><span class="p">,</span> <span class="n">k8sObjectName</span><span class="o">:</span> <span class="s">"cluster-autoscaler"</span><span class="p">,</span> <span class="n">kubeContext</span><span class="o">:</span> <span class="s">"test-context"</span><span class="p">,</span> <span class="n">namespace</span><span class="o">:</span> <span class="s">"kube-system"</span><span class="p">,</span>
<span class="n">deployment</span><span class="o">:</span> <span class="o">&</span><span class="n">appsv1</span><span class="o">.</span><span class="n">Deployment</span><span class="p">{}},</span>
<span class="n">output</span><span class="o">:</span> <span class="s">""</span><span class="p">,</span>
<span class="n">err</span><span class="o">:</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="s">"Deployment cluster-autoscaler in namespace kube-system not found</span><span class="se">\n</span><span class="s">"</span><span class="p">),</span>
<span class="p">},</span>
<span class="p">}</span>
<span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">tt</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">tests</span> <span class="p">{</span>
<span class="n">t</span><span class="o">.</span><span class="n">Run</span><span class="p">(</span><span class="n">tt</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">t</span> <span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span> <span class="p">{</span>
<span class="n">client</span> <span class="o">:=</span> <span class="n">fake</span><span class="o">.</span><span class="n">NewSimpleClientset</span><span class="p">(</span><span class="n">tt</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">deployment</span><span class="p">)</span>
<span class="n">got</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">GetContainerImageForK8sObject</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">tt</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">k8sObjectName</span><span class="p">,</span> <span class="n">tt</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">k8sObject</span><span class="p">,</span> <span class="n">tt</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">namespace</span><span class="p">)</span>
<span class="n">assert</span><span class="o">.</span><span class="n">Equal</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">tt</span><span class="o">.</span><span class="n">output</span><span class="p">,</span> <span class="n">got</span><span class="p">)</span>
<span class="n">assert</span><span class="o">.</span><span class="n">Equal</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">tt</span><span class="o">.</span><span class="n">err</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A full example of the same which I wrote is here where <a href="https://github.com/deliveryhero/k8s-cluster-upgrade-tool/blob/d19b863c9daec01e3f5378ae7a32fb3bf94e86bd/internal/api/k8s/cluster.go#L111"><code class="language-plaintext highlighter-rouge">GetContainerImageForK8sObject()</code></a> is the function under test, which takes the client as a dependency and then in the test case we would use the fake client <a href="https://github.com/deliveryhero/k8s-cluster-upgrade-tool/blob/v0.4.0/internal/api/k8s/cluster_test.go#L44-L100">here like this</a></p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://github.com/kubernetes/client-go">https://github.com/kubernetes/client-go</a></li>
<li><a href="https://github.com/deliveryhero/k8s-cluster-upgrade-tool">https://github.com/deliveryhero/k8s-cluster-upgrade-tool</a></li>
<li><a href="https://pkg.go.dev/k8s.io/client-go/kubernetes/fake">https://pkg.go.dev/k8s.io/client-go/kubernetes/fake</a></li>
<li><a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L613">BuildConfigFromFlags()</a></li>
<li><a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L134">ClientConfig()</a></li>
<li><a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L462">getContextName()</a></li>
<li><a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/client_config.go#L144">getContext()</a></li>
<li><a href="https://github.com/kubernetes/client-go/blob/1110612dc6e599ae817abbcb762c7c5e87e99a51/tools/clientcmd/overrides.go#L30">ConfigOverrides</a></li>
</ul>
Moving over to www.tasdikrahman.com from tasdikrahman.me
2022-09-17T00:00:00+00:00
https://www.tasdikrahman.com/2022/09/17/moving-over-to-www-tasdikrahman-com-from-tasdikrahman-me
<p>I have been writing over at this blog since mid 2015 and not much has changed over these years on this blog, the same theme, the same static website generater, the same color scheme.</p>
<h2 id="why-did-i-end-up-moving">Why did I end up moving?</h2>
<p>The .me domain name, from what I could gather, the registry which is operating it has access to it only till 2023, not that I am anticipating anything out of the blue, but I felt it just makes sense long term to move to .com.</p>
<p>Plus, the .me domain was with me since it came free along with the github education pack back in the days in college, which is why my domain tasdikrahman.me existed in the first place, I never ended up moving away from it at that time.</p>
<h2 id="what-did-i-end-up-doing">What did I end up doing</h2>
<h3 id="add-redirect-on-the-old-page">Add redirect on the old page</h3>
<p>Looking around, since the page was statically generated, I found it just easier to add a block</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><meta</span> <span class="na">http-equiv=</span><span class="s">"Refresh"</span> <span class="na">content=</span><span class="s">"0; url='https://www.tasdikrahman.com'"</span> <span class="nt">/></span>
</code></pre></div></div>
<p>in the <code class="language-plaintext highlighter-rouge">head.html</code> for the older website which I was having, since each generated html page would need to have it, ended up adding it in <code class="language-plaintext highlighter-rouge">head.html</code> <a href="https://github.com/tasdikrahman/tasdikrahman.me/commit/f67b2110789427bc9f43b073c66f796512e83737">here</a></p>
<p>There are more ways to do the same, one of them being https://about.txtdirect.org/hosted/, which would make use to TXT records, and act as the middle layer, to redirect back to wherever you want it to go.</p>
<p>I didn’t want to make use of an external service/entity at this point unless really required, hence didn’t end up going in that direction, although if you have other suggestions, would love to hear about them.</p>
<h3 id="adding-redirect-for-tasdikrahmancom-to-redirect-to-wwwtasdikrahmancom">Adding redirect for tasdikrahman.com to redirect to www.tasdikrahman.com</h3>
<p>This was simple and easy, adding a temporaty redirect on the domain registrar (google domains in this case) was simple and straight forward.</p>
<p>Enabling SSL on for the same again super simple, as github pages gives the option to enable it straight on the settings of the repo from where the static website is being generated from.</p>
<h2 id="ending-notes">Ending notes</h2>
<p>I have always loved the simplicity of static website generators, the fact that I haven’t changed anything for so long, and the add markdown, commit changes, git push model reducing the barrier to writing for me is a huge plus for me.</p>
<p>Github pages exposing the github actions used to build and deploy the pages also makes it easier in case I want to move to another static website generator at some point. Plus this made me realize that the blog has grown over to <a href="https://github.com/tasdikrahman/tasdikrahman.com/actions/runs/3028630771">~100MB</a> as of now.</p>
Stubbing and few other testing tidbits for python
2022-04-15T00:00:00+00:00
https://www.tasdikrahman.com/2022/04/15/stubbing-and-a-few-other-testing-tidbits-for-python
<p>It’s been sometime since I wrote some python, and ended doing a bit of testing for a couple of routines which I ended up implementing. This post is more about me just condensing those ideas for python and how to do it in python, but the ideas are also a carryover from my other testing experiences, while using other languages and how my ideas for testing have progressed over time comparing some testing which I had done in some projects some years back. You can find a couple of more posts under <a href="https://www.tasdikrahman.com/blog/tag/testing/">https://www.tasdikrahman.com/blog/tag/testing/</a> where I have delved more into these topics.</p>
<p>As always, I will for sure look at this post at some point and notice improvements as my thoughts on testing progress and mature.</p>
<h2 id="what-about-non-deterministic-tests">What about non-deterministic tests</h2>
<p>I picked this piece of code from an old project which I worked on long back in college for this bit, if you look at the following block</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># picked from here
# https://github.com/tasdikrahman/plino/blob/713ad80524bb4038cb08475b299b02cca3fe7feb/tests/test_plino_app_api_response.py#L37
</span> <span class="k">def</span> <span class="nf">test_api_spam_email_text</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""
Unit test to verify the 200 response code and the correct email_class
returned by API when a spam email text is passed
"""</span>
<span class="n">payload</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'email_text'</span><span class="p">:</span> <span class="n">SPAM_TEXT</span>
<span class="p">}</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">'content-type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">}</span>
<span class="n">response</span> <span class="o">=</span> \
<span class="n">requests</span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">api_url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">payload</span><span class="p">),</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">content</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">response</span><span class="p">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span>
<span class="k">assert</span> <span class="n">r</span><span class="p">[</span><span class="s">'email_class'</span><span class="p">]</span> <span class="o">==</span> <span class="s">'spam'</span>
</code></pre></div></div>
<p>This for me, if I look at it now is more noise than signal, as the probability of it landing up in being classified to something which we are asserting it to, is probabilistic in nature, I would rather have this as a <a href="https://en.wikipedia.org/wiki/Test_oracle">test oracle</a> for me to be able to know what is going on for this case at least.</p>
<p>Another thing to note here in this spec if that, there is a scope creep happening here, I am trying to do two things at the same time. One being, trying to test the route <code class="language-plaintext highlighter-rouge">api/v1/classify/</code>, and check for the response code for a successfull response and the second thing being, I am also trying to test the domain specific implementation of domain logic. Mixing these two don’t really make sense at this point.</p>
<p>What I would have rather done at this point, is to inject the dependency of the domain specific logic to return the response for which I was testing the api response code for, making this spec deterministic and reducing the noise.</p>
<p>This would have also reduced the extra test behaviour which was being tested here in this case, which was unnecessary. Plus the underlying domain logic being tested was the classifier in this case, which would then be testing the 3rd party codebase itself, which is not required. What we would want to rather do is wrap our business logic around the responses which the 3rd party flow can give.</p>
<h2 id="stubbing-responses-in-this-case-stubbing-a-method-which-receives-stdin">Stubbing responses, in this case stubbing a method which receives STDIN</h2>
<p>Will pluck out the irrelevant details from this spec I wrote for <a href="https://docs.python.org/3/library/fileinput.html"><code class="language-plaintext highlighter-rouge">fileinput.input()</code></a>, the context was that a method was using fileinput.input() to read from STDIN, and we needed to test the original method, without actually waiting for STDIN in our test spec runner.</p>
<p>Here’s a snippet describing changing the design of the implementation to prevent the call of fileinput.input() while the test run.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># initial design
</span><span class="kn">import</span> <span class="nn">fileinput</span>
<span class="k">class</span> <span class="nc">IO</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">read_stdin</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""
read_stdin will read the STDIN and process the data received
:returns: list of sentences read line by line
"""</span>
<span class="n">lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">cleaned_lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">std_in</span> <span class="o">=</span> <span class="n">fileinput</span><span class="p">.</span><span class="nb">input</span><span class="p">()</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">std_in</span><span class="p">:</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="c1"># strip newlines
</span> <span class="n">cleaned_lines</span> <span class="o">+=</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">]</span>
<span class="c1"># remove empty strings
</span> <span class="k">return</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">cleaned_lines</span> <span class="k">if</span> <span class="n">x</span><span class="p">.</span><span class="n">strip</span><span class="p">()]</span>
</code></pre></div></div>
<p>Now in our test spec, if we wanted to call this method and assert for list output which it returned, we would be stuck with the STDIN IO wait time here.</p>
<p>Rather adding an interface on top of this behaviour would help us further in stubbing that response out which we get from fileinput.input()</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">fileinput</span>
<span class="k">class</span> <span class="nc">IO</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">read_stdin</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">"""
read_stdin will read the STDIN and process the data received
:returns: list of sentences read line by line
"""</span>
<span class="n">lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">cleaned_lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">std_in</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">get_stdin</span><span class="p">()</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">std_in</span><span class="p">:</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="c1"># strip newlines
</span> <span class="n">cleaned_lines</span> <span class="o">+=</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">]</span>
<span class="c1"># remove empty strings
</span> <span class="k">return</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">cleaned_lines</span> <span class="k">if</span> <span class="n">x</span><span class="p">.</span><span class="n">strip</span><span class="p">()]</span>
<span class="o">@</span><span class="nb">staticmethod</span>
<span class="k">def</span> <span class="nf">get_stdin</span><span class="p">():</span>
<span class="k">return</span> <span class="n">fileinput</span><span class="p">.</span><span class="nb">input</span><span class="p">()</span>
</code></pre></div></div>
<p>Now we can stub this method in our spec, since we already knew the behaviour of the stubbed method and what it would give us, we added the response value for it for our spec, effectively replacing an actual call. It’s interesting to see this in the decorator syntax provided and looks quite clean to read.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">fileinput</span>
<span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">TestCase</span>
<span class="kn">from</span> <span class="nn">mock</span> <span class="kn">import</span> <span class="n">patch</span>
<span class="k">class</span> <span class="nc">TestReadStdin</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="o">@</span><span class="n">patch</span><span class="p">.</span><span class="nb">object</span><span class="p">(</span><span class="n">io</span><span class="p">.</span><span class="n">IO</span><span class="p">,</span> <span class="s">"get_stdin"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_read_stdin</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">stub_get_stdin</span><span class="p">):</span>
<span class="n">content</span> <span class="o">=</span> <span class="s">"""I could not help it, but I began to feel suspicious of this. At any rate, I made up my mind that if it so turned out that we should sleep together, he must undress and get into bed before I did.
Supper over, the company went back to the bar-room, when, knowing not what else to do with myself, I resolved to spend the rest of the evening as a looker on."""</span>
<span class="n">want</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">"I could not help it, but I began to feel suspicious of this. At any rate, I made up my mind that if it so turned out that we should sleep together, he must undress and get into bed before I did."</span><span class="p">,</span>
<span class="s">"Supper over, the company went back to the bar-room, when, knowing not what else to do with myself, I resolved to spend the rest of the evening as a looker on."</span><span class="p">,</span>
<span class="p">]</span>
<span class="c1"># create the temp file
</span> <span class="k">with</span> <span class="n">TestFileContent</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="k">as</span> <span class="n">valid_file</span><span class="p">:</span>
<span class="n">stub_get_stdin</span><span class="p">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="n">fileinput</span><span class="p">.</span><span class="nb">input</span><span class="p">(</span><span class="n">files</span><span class="o">=</span><span class="n">valid_file</span><span class="p">.</span><span class="n">filename</span><span class="p">)</span>
<span class="n">io_obj</span> <span class="o">=</span> <span class="n">io</span><span class="p">.</span><span class="n">IO</span><span class="p">()</span>
<span class="n">got</span> <span class="o">=</span> <span class="n">io_obj</span><span class="p">.</span><span class="n">read_stdin</span><span class="p">()</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">want</span><span class="p">,</span> <span class="n">got</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="testing-for-stdout">Testing for STDOUT</h2>
<p>It’s of value to test out the STDOUT being received for certain cases. For example</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Class</span> <span class="n">Printer</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">print_this</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s">"{0} - {1}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="s">" "</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">key</span><span class="p">),</span> <span class="n">value</span><span class="p">))</span>
</code></pre></div></div>
<p>spec for the same will look like</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">TestCase</span>
<span class="kn">import</span> <span class="nn">io</span>
<span class="kn">import</span> <span class="nn">unittest.mock</span>
<span class="c1"># import Printer
</span>
<span class="k">class</span> <span class="nc">TestPrinter</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="o">@</span><span class="n">unittest</span><span class="p">.</span><span class="n">mock</span><span class="p">.</span><span class="n">patch</span><span class="p">(</span><span class="s">"sys.stdout"</span><span class="p">,</span> <span class="n">new_callable</span><span class="o">=</span><span class="n">io</span><span class="p">.</span><span class="n">StringIO</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">assert_stdout</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">test_input_a</span><span class="p">,</span> <span class="n">test_input_b</span><span class="p">,</span> <span class="n">expected_output</span><span class="p">,</span> <span class="n">mock_stdout</span><span class="p">):</span>
<span class="n">print_obj</span> <span class="o">=</span> <span class="n">Printer</span><span class="p">()</span>
<span class="n">print_obj</span><span class="p">.</span><span class="n">print_this</span><span class="p">(</span><span class="n">test_input_a</span><span class="p">,</span> <span class="n">test_input_b</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">mock_stdout</span><span class="p">.</span><span class="n">getvalue</span><span class="p">(),</span> <span class="n">expected_output</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_print_ranked</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">test_input_a</span> <span class="o">=</span> <span class="n">foo</span>
<span class="n">test_input_b</span> <span class="o">=</span> <span class="n">baz</span>
<span class="n">want</span> <span class="o">=</span> <span class="s">"foo - baz"</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assert_stdout</span><span class="p">(</span><span class="n">test_input_a</span><span class="p">,</span> <span class="n">test_input_b</span><span class="p">,</span> <span class="n">want</span><span class="p">)</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">mock_stdout</code> arg is passed automatically by the <code class="language-plaintext highlighter-rouge">unittest.mock.patch</code> decorator to the <code class="language-plaintext highlighter-rouge">assert_stdout</code> method.</p>
<h2 id="io-related-to-files">IO related to files</h2>
<p>python already provides a great interface to creating temporary files and deleting them after their use case has been achieved, as compared to golang for example.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Is there a better way than to wrap the os.Remove(filename) with a defer, for a file created via io/ioutil in <a href="https://twitter.com/golang?ref_src=twsrc%5Etfw">@golang</a>?</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1512697739196477440?ref_src=twsrc%5Etfw">April 9, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>The following block would add the content to the file straight up while creating the context block for the file.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">TestFileContent</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">content</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="nb">file</span> <span class="o">=</span> <span class="n">tempfile</span><span class="p">.</span><span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="n">mode</span><span class="o">=</span><span class="s">"w"</span><span class="p">,</span> <span class="n">delete</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="nb">file</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">filename</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="nb">file</span><span class="p">.</span><span class="n">name</span>
<span class="k">def</span> <span class="nf">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span>
<span class="k">def</span> <span class="nf">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">type</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">traceback</span><span class="p">):</span>
<span class="n">os</span><span class="p">.</span><span class="n">unlink</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">filename</span><span class="p">)</span>
</code></pre></div></div>
<p>Now this can be simply used like this, whereas the name of the file can be plucked out via the filename attribute here.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">TestReadFiles</span><span class="p">(</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_read_files_read_single_file</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">content</span> <span class="o">=</span> <span class="s">"""
Supper over, the company went back to the bar-room, when, knowing not what else to do with myself, I resolved to spend the rest of the evening as a looker on."""</span>
<span class="n">want</span> <span class="o">=</span> <span class="s">"test_output"</span> <span class="p">]</span>
<span class="k">with</span> <span class="n">TestFileContent</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="k">as</span> <span class="n">valid_file</span><span class="p">:</span>
<span class="n">got</span> <span class="o">=</span> <span class="n">test_method</span><span class="p">([</span><span class="n">valid_file</span><span class="p">.</span><span class="n">filename</span><span class="p">])</span>
<span class="bp">self</span><span class="p">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">want</span><span class="p">,</span> <span class="n">got</span><span class="p">)</span>
</code></pre></div></div>
<p>If you wanted to create create multiple temporary files in the same context for simpler cleanup.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">with</span> <span class="n">TestFileContent</span><span class="p">(</span><span class="n">content_file_a</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_a</span><span class="p">,</span> <span class="n">TestFileContent</span><span class="p">(</span>
<span class="n">content_file_b</span>
<span class="p">)</span> <span class="k">as</span> <span class="n">file_b</span><span class="p">:</span>
<span class="p">...</span>
<span class="p">...</span>
</code></pre></div></div>
<p>Asserting for file not read errors</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">test_read_files_file_not_found</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="n">assertRaises</span><span class="p">(</span><span class="nb">FileNotFoundError</span><span class="p">):</span>
<span class="n">io_obj</span> <span class="o">=</span> <span class="n">io</span><span class="p">.</span><span class="n">IO</span><span class="p">()</span>
<span class="n">io_obj</span><span class="p">.</span><span class="n">read_files</span><span class="p">([</span><span class="s">"non-existent-file.txt"</span><span class="p">])</span>
</code></pre></div></div>
<p>In case you would like to test the behaviour when the user is not allowed to read the file content due to permission error</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">test_read_files_file_read_permission_error</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">content</span> <span class="o">=</span> <span class="s">"""foo"""</span>
<span class="k">with</span> <span class="n">TestFileContent</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> <span class="k">as</span> <span class="n">valid_file</span><span class="p">:</span>
<span class="n">io_obj</span> <span class="o">=</span> <span class="n">io</span><span class="p">.</span><span class="n">IO</span><span class="p">()</span>
<span class="c1"># make file not readable to user
</span> <span class="n">os</span><span class="p">.</span><span class="n">chmod</span><span class="p">(</span><span class="n">valid_file</span><span class="p">.</span><span class="n">filename</span><span class="p">,</span> <span class="mo">0o0230</span><span class="p">)</span>
<span class="k">with</span> <span class="bp">self</span><span class="p">.</span><span class="n">assertRaises</span><span class="p">(</span><span class="n">PermissionError</span><span class="p">):</span>
<span class="n">io_obj</span><span class="p">.</span><span class="n">read_files</span><span class="p">([</span><span class="n">valid_file</span><span class="p">.</span><span class="n">filename</span><span class="p">])</span>
</code></pre></div></div>
<p>Will most likely add more references for myself here or in another post.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://stackoverflow.com/a/46307456">https://stackoverflow.com/a/46307456</a></li>
<li><a href="https://docs.python.org/3/library/fileinput.html">https://docs.python.org/3/library/fileinput.html</a></li>
<li><a href="https://stackoverflow.com/a/54053967">https://stackoverflow.com/a/54053967</a></li>
</ul>
spf13/cobra not respecting mandatory flags as part of Prerun
2022-02-28T00:00:00+00:00
https://www.tasdikrahman.com/2022/02/28/workaround-for-spf13-cobra-prerun-not-respecting-mandatory-flags
<p>Just a continuation of the tweet, adding into small snippets for context.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Came across this issue where spf13/cobra would not work for mandatory flags set, when PreRun is set for a command while building a cli tool <a href="https://t.co/GmULRFrFfm">https://t.co/GmULRFrFfm</a> (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1498355645392818178?ref_src=twsrc%5Etfw">February 28, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="context">Context</h2>
<p>When you have both the <code class="language-plaintext highlighter-rouge">PreRun</code> and <code class="language-plaintext highlighter-rouge">Run</code> directives, and the mandatory flag present, which you expect to run before the <code class="language-plaintext highlighter-rouge">PreRun</code> directives mentioned, it will not be respected. This post is just a small nudge to prevent someone from trying to achieve the same as the same hasn’t been documented on cobra.dev side so far (in case nothing has been missed)</p>
<div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// sample snippet from https://cobra.dev/#prerun-and-postrun-hooks</span>
<span class="o">...</span>
<span class="o">...</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="k">var</span> <span class="n">rootCmd</span> <span class="o">=</span> <span class="o">&</span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="p">{</span>
<span class="n">Use</span><span class="o">:</span> <span class="s">"root [sub]"</span><span class="p">,</span>
<span class="n">Short</span><span class="o">:</span> <span class="s">"My root command"</span><span class="p">,</span>
<span class="n">PreRun</span><span class="o">:</span> <span class="k">func</span><span class="p">(</span><span class="n">cmd</span> <span class="o">*</span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="p">,</span> <span class="n">args</span> <span class="p">[]</span><span class="kt">string</span><span class="p">)</span> <span class="p">{</span>
<span class="c">// Logic for PreRun</span>
<span class="p">},</span>
<span class="n">Run</span><span class="o">:</span> <span class="k">func</span><span class="p">(</span><span class="n">cmd</span> <span class="o">*</span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="p">,</span> <span class="n">args</span> <span class="p">[]</span><span class="kt">string</span><span class="p">)</span> <span class="p">{</span>
<span class="c">// Logic for Run</span>
<span class="p">},</span>
<span class="o">...</span>
<span class="o">...</span>
</code></pre></div></div>
<p>Now in case you have added mandatory flags inside your <code class="language-plaintext highlighter-rouge">init()</code> function</p>
<div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">...</span>
<span class="k">func</span> <span class="n">init</span><span class="p">()</span> <span class="p">{</span>
<span class="o">...</span>
<span class="c">// picked from https://cobra.dev/#required-flags</span>
<span class="n">rootCmd</span><span class="o">.</span><span class="n">Flags</span><span class="p">()</span><span class="o">.</span><span class="n">StringVarP</span><span class="p">(</span><span class="o">&</span><span class="n">Region</span><span class="p">,</span> <span class="s">"region"</span><span class="p">,</span> <span class="s">"r"</span><span class="p">,</span> <span class="s">""</span><span class="p">,</span> <span class="s">"AWS region (required)"</span><span class="p">)</span>
<span class="n">rootCmd</span><span class="o">.</span><span class="n">MarkFlagRequired</span><span class="p">(</span><span class="s">"region"</span><span class="p">)</span>
<span class="o">...</span>
<span class="p">}</span>
<span class="o">...</span>
</code></pre></div></div>
<p>Now if you run the cli without passing the required flag or <code class="language-plaintext highlighter-rouge">r</code> when running the cli, the routines inside <code class="language-plaintext highlighter-rouge">PreRun</code> don’t get run, and this flag is not respected anymore.</p>
<h2 id="workaround-1">Workaround 1</h2>
<p>Just move the flow inside the <code class="language-plaintext highlighter-rouge">PreRun</code> to <code class="language-plaintext highlighter-rouge">Run</code>, as there is no workaround as of now, even the same has been done by some other projects like <a href="https://github.com/keptn/keptn/issues/2729">keptn</a>, where they have done the same. Agreed this bloats the <code class="language-plaintext highlighter-rouge">Run</code>, directive, but until there is a better solution, it’s better than not having any validation and repeating what the framework has already done for you.</p>
<h2 id="workaround-2">Workaround 2</h2>
<p>You can start using the <code class="language-plaintext highlighter-rouge">cobra.ExactArgs()</code> instead of using flags, this solution looks a bit confusing for the user using the CLI though, as there would be no named arguments and the CLI user has to then actually start remembering which positional arguments they need to pass to the CLI and which position is for what, leading to a CLI which requires to actually have a mental model</p>
<h2 id="comparison-of-the-two-workarounds-for-the-cli">Comparison of the two workarounds for the cli</h2>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// <span class="k">case</span> of using the mandatory flag withouth any PreRun and having the valition inside Run directive <span class="k">for </span>the user to pass the region
<span class="nv">$ </span>my_cli foo-command <span class="nt">-r</span><span class="o">=</span>region-foo
// using the positional argument
<span class="nv">$ </span>my_cli foo-command region-foo
</code></pre></div></div>
<h2 id="ending-notes">Ending notes</h2>
<p>Given these two alternatives, I would prefer the 1st workaround simply because the CLI user doesn’t have to then remember what does the positional arguments stand for and we can leverage the framework to direct which flag is standing for what attribute and what do they need to pass further.</p>
<p>I didn’t find anything, the last open ticket on <a href="https://github.com/spf13/cobra/issues/655">this</a> is still open, I ended up <a href="https://github.com/spf13/cobra/issues/655#issuecomment-1054509187">enquiring</a> in case this is by design, in case I have missed something. Will check back and update here in case there is an update.</p>
<p>There was an old <a href="https://github.com/spf13/cobra/issues/206">issue</a> which talks about the introduction of the mandatory flags, but doesn’t go into the <code class="language-plaintext highlighter-rouge">PreRun</code> directives.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://github.com/keptn/keptn/issues/2729">https://github.com/keptn/keptn/issues/2729</a></li>
<li><a href="https://github.com/spf13/cobra/issues/655">https://github.com/spf13/cobra/issues/655</a></li>
<li><a href="https://github.com/spf13/cobra/issues/655#issuecomment-1054509187">https://github.com/spf13/cobra/issues/655#issuecomment-1054509187</a></li>
</ul>
<h2 id="credits">Credits</h2>
<p>Image credits for the post https://cobra.dev/</p>
Evolution of support for infrastructure teams
2022-02-21T00:00:00+00:00
https://www.tasdikrahman.com/2022/02/21/evolution-of-support-for-infrastructure-teams
<h2 id="context">Context</h2>
<p>As time has progressed, I have been part of teams of different sizes in terms of org size as well as the team sizes which I have been part of. This post is a conglomeration of the ideas I have picked up, things which have worked out/which haven’t and mental models developed as being a part of such infrastructure teams and growing with them.</p>
<p>Another thing noticed over time, would be the amount of adhoc work being slightly higher than other engineering teams out there, hence the difference in structure on how to handle it as we will see over the course of this post.</p>
<p>This post condenses the ideas presented here.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">A little late to the party but here are a few ways, infrastructure teams could function during the organization growth phases and different engineering team sizes over time (1/n) 🧵 <a href="https://t.co/wUC7ujyaU3">https://t.co/wUC7ujyaU3</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1495881281745440772?ref_src=twsrc%5Etfw">February 21, 2022</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="before-we-start">Before we start</h2>
<p>I am deliberately not putting out number of engineers here to fit a case because you will start feeling the toil and support request influx eventually, leading you to adopt and try out different models of support over time as this also depends on things apart from team size.</p>
<h2 id="initial-days">Initial days</h2>
<p>For starters, for a new infrastructure team, you will notice the team getting adhoc requests on DM’s/Slack channels/phone calls too maybe and other mediums of communication which you use. This is workable only up till the point where the respective infrastructure team folks don’t get tired of responding to requests on the above mediums, which is bound to happen, as the distribution is random at best.</p>
<p>But if we look at it from the engineering teams perspective this is normal since there is no clear format/expectation set, which is what the infrastructure team needs to first do/would already be doing in some ways already. For the former part, the said person, receiving a request may then point it to the other team members on the standup that foo request was made and discuss it with others or if the task was too small maybe they already implemented it already before the next day.</p>
<p>The first thing which we can do in such a case would be, to ask folks requesting support to request for it on a respective slack channel and mark it as feature/bug fix/emergency request/query. If there’s a ticketing system/kanban board for the infra team, ask them to create the ticket/card there and let someone from the infra team(possibly the manager/lead) route such requests to the infra team members depending on their prio and availability.</p>
<p>This lets you solve a couple of things, the prio problem, which then gets offloaded to a single person rather than the whole team, and the problem of discovery of what were the support requests which the team got/solved.</p>
<p>One thing to note here that, there is most likely no incentive for the infra team member to solve tickets, hence I have only seen this model work when there is a gentle bump each time from someone who is keeping track of the prio of the tickets and their status.</p>
<h2 id="as-the-team-size-grows">As the team size grows</h2>
<p>The team sizes grow, along with members in your team also growing, but the toil may also have increased over time, which could be a symptom of a couple of things. If you have been tracking the support requests which you have gotten, by this time you would have noticed what were the kind of requests which you most get frequently and how long do such requests take. The idea is to see what requests does your team get and how much time are you spending on such things for starters. After this, you would have a decent picture of requests/problems which your team is getting and you can focus on reducing the influx or requests by either.</p>
<p>Documentation to fix knowledge gaps for the team, helping them gain context and close off queries quickly. This also helps distribute knowledge inside the team. A general rule of thumb could be if you do it twice, write it down.</p>
<p>Automation being added to do repeated tasks, which have fairly predictable branch off’s from the happy paths, which can be handled by the automation for those cases when it deviates from the happy path.</p>
<p>This can be the next step of documentation, which supplements the knowledge by putting the operational knowledge in a tool. But remember that prematurely writing the automation is not any better than not having any automation.</p>
<p>All code is liability, so be careful on what you write the automation in, it should be something which can be maintained by at least a couple of people in the team, to help de-risk the automation becoming a fragile codebase which the team is afraid to touch.</p>
<h2 id="further-growth-and-division-into-sub-teams">Further growth and division into sub-teams</h2>
<p>After a point of growth in your team, you will start noticing the larger infrastructure team to branch off and have separate sub teams for specific interest groups which the team works on. For example, observability, release engg, infrastructure, security, dev experience etc start getting formed as the team’s requirement for more people to focus on specific problems arise from the growth in the engineering team.</p>
<p>At this point, the original problem of toil still remains and will remain for each sub-team to handle, the only difference here will be that, each ticket will then have a specific team to get routed to.</p>
<p>It can also become a point of contention on what parts are to be owned by which sub-team, hence preferably, the ownership of specific components should be pre-decided for each sub-team, apart from the obvious ownership of components. This helps resolve issues/confusion arising from who would own a certain ticket, the back and forth hence causing a possible fallthrough in SLA.</p>
<p>On SLA’s, the teams would need to decide on the expectations on what is a good turn around time for a ticket requested, based on the priority on which it has been raised.</p>
<p>There will be cases when requests raised on priority P0/P1 might be P2/P3. A good rule of thumb is to have anything affecting affecting customers/bleeding money/security incident as P0 and needs immediate attention. The other prio groups can be discussed and decided upon. SLAs/tracking of requests/routing of tickets being at least have been addressed in this first pass, let’s look at SLA’s.</p>
<p>You might notice after a certain org engg team size and each sub-team having it’s own sub section in the Quarters planning doc, that you might be falling short on SLA’s for the support tickets turn around time.</p>
<p>It could be that there is automation missing, docs missing, or simply could be that the amount of support requests is simply big enough for the individual sub teams to be routed just via the TL/manager.</p>
<p>As they would also be distributing the request over to someone initially depending on their familiarity with who understands which stacks, but ideally this distribution in the sub team should be random at best as everyone should be distributed these requests equally.</p>
<p>If the request workload is still increasing, it might be time to introduce rotations for support inside the sub teams, having dedicated people from the sub teams to look and work on issues for the sprints duration and then rotating away afterwards</p>
<p>This helps in a couple of ways, for starters, helps prevent burnout for the team members as support becomes democratized within team. Since not everyone gets support tickets by default, the others can focus on strategic problems to be solved. The idea should be that, apart from toil work. There should be ample amount of time left for strategic work, which helps reduce toil work and improves dev experience, helping create good abstractions</p>
<h2 id="further-improving-productivity-inside-the-sub-teams">Further improving productivity inside the sub-teams</h2>
<p>You could further go into having dedicated folks for support for each teams to having dedicated folks for handling support for infrastructure sub-team engineering support. This will not immediately start showing any positive impact as with every new member in the team, it takes some time for the impact to come across, but having an L1 support structure pays back in time for the overall team</p>
<p>This L1 support team can be the first line of defense for anything related to infrastructure queries for all the sub-teams inside</p>
<p>Over time with pairing and knowledge transfers, the idea would be for the L1 support team to handle some level of queries/providing solutions with some/no amount of support from the sub-teams. As time goes, these L1 folks would have enough context to even start working inside the specific sub-teams and absorbed there if the teams would like to, creating a funnel of engineers joining the team via L1 -> sub-team member and so on and so forth.</p>
<p>At this stage your infrastructure team could very well be said to be in a mature stage. And would have been able to keep support request SLAs to a favorable number along with working on strategic things.</p>
<h2 id="final-thoughts">Final thoughts</h2>
<p>As always, the main thing to note here would be reduction of toil as time goes by and keeping track of how much time is one spending on tickets/support requests, everyother solutioning follows this quest of reducing toil and a side outcome as part of it.</p>
<p>Would love to hear, how you folks handle support requests and adhoc work in your org</p>
<h2 id="references">References</h2>
<p>Cover image by Harald Süpfle, CC BY-SA 2.5 <a href="https://creativecommons.org/licenses/by-sa/2.5">https://creativecommons.org/licenses/by-sa/2.5</a>, via Wikimedia Commons, https://commons.wikimedia.org/wiki/File:Webervogelnst_Auoblodge.JPG</p>
Building the VM creation API for the org
2021-06-20T00:00:00+00:00
https://www.tasdikrahman.com/2021/06/20/building-the-vm-creation-api-for-the-org
<p>Over the years, there have been a lot of changes in the way, people create their virtual machines in their cloud environment.</p>
<p>At a very primitive level, one would simply go about doing it via the cloud provider console. A couple of clicks and lo and behold.</p>
<p>At a larger scale, people end up using automation to create these Virtual machines in the way they want them to be, given the manual nature of work would just start becoming a bottleneck in scaling quickly when required otherwise.</p>
<p>So how did we go about it?</p>
<h2 id="existing-solution">Existing solution</h2>
<p>I have spoken a bit about our current automation using <a href="https://github.com/gojek/proctor">proctor</a>,</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/mE1JZKMhnNs" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>I will not delve more into that, but in gist, we had a way present in which developers could simply demand a VM getting created via a cli interface.</p>
<p>As time went by, the automations to create different variations of virtual machines based on language stack kept getting added, helping developers create Virtual machines on demand for their use cases.</p>
<p>This post is about how we ended up making the building block for a lot of automation around Virtual machine creation orchestration inside the org, as part of the platform team.</p>
<h2 id="what-is-the-need-for-a-vm-creation-api-in-the-first-place">What is the need for a VM creation API in the first place?</h2>
<p>Yes, true. Everything is working fine and dandy. People are able to create VM’s on demand whenever they want and however they would like to, via proctor and everyone is happy so far. What’s the issue then?</p>
<p>We come along with a couple of deprecation plans for the virtual machine infrastructure, where we have to deprecate whole fleets of VM’s for applications which were running on 16.04, which got deprecated in April this year.</p>
<p>It started with adding support for 20.04 to all the automation present in the org, to be able to support creating 20.04 VM’s via proctorˀ. It meant adding support to all the proctor automation scripts which we had, testing them out and so on and so forth.</p>
<p>The part to note here is that the automation’s interface was still the same via the command line tool to create VM’s for end users.</p>
<h2 id="what-we-ended-up-doing">What we ended up doing</h2>
<p>We already had a service which we had written to create kubernetes clusters on demand. Rughly, each kubernetes cluster would simply be a resource for this service, where it would store all the cluster related metadata.</p>
<p>Given the programmatic way of creating the kubernetes clusters (although I will share more about the challenges of this approach of creating and managing k8s clusters in another post), we wanted to add another resource on top of our modelling for this service. The service would at the end just call proctor and send it POST calls to create VM’s and have a callback link which this service would poll at the end of the day to check whether the given resources were created at the end of the day or not.</p>
<p>One difference from proctor, which would come here is that, one could ask for multiple VM’s getting created for the application which the service was trying to create VM’s for.</p>
<p>We also ended up changing the way the resources were named, to improve upon our existing way of naming resources (more on this later)</p>
<p>There’s now a central place where we are storing, which applications are running which version of VM’s and how many of them, of what language stack and what OS flavor and other additional metadata.</p>
<p>We also ended up adding a client for it, which people could use to create VM’s of their choice of demand, adding all the necessary helper methods (polling, checking resource status etc.)</p>
<h2 id="user-flow">User Flow</h2>
<center><img src="/content/images/2021/06/24BDDA89-7EF4-4E21-B244-919D364A386D.jpeg" /></center>
<p>1) The user sends a <code class="language-plaintext highlighter-rouge">POST</code> call to orchestration service to the VM creation route, sending all the necessary information in the <code class="language-plaintext highlighter-rouge">POST</code> body to create the VM/VM’s for their application. Information like, which application, the environment, the type of VM to pick, the OS version etc</p>
<p>2) Orchestration service validates the request body and presence of certain fields in the request body, which are required for creation of the VM. eg: environment, language stack etc. It would also send a response back to the client, which would include the information like the VM names etc, which would be something which the user would find useful. Given the async nature of the response, we store the state of the creation of the VMs for each resource. The client can query the resource and the response body would also send the state details of the resource.</p>
<p>3) Orchestration service constructs the information required to send to proctor daemon to call the right proctor job, with the correct request body, sending multiple requests to proctor in case multiple VM’s are requested for creation.</p>
<p>4) Proctor daemon runs the automation to create the VM in a background job to provision and configure the VM.</p>
<p>5) Orchestration service polls proctor daemon for the job which was scheduled to create the VM, with a default backoff present.</p>
<p>6 & 7) proctor job completes, if the poll gets a response within the appropriate time, the status of the job is saved in orchestration service, which decides the final status of the resource in orchestration service DB too.</p>
<p>The user can at this point, make a <code class="language-plaintext highlighter-rouge">GET</code> call on the resource URI, to see the updated resource status to check what’s the current state of the state machine.</p>
<h2 id="how-has-it-been-used-so-far">How has it been used so far</h2>
<p>The very first thing, this API got used for was to automatically create our version of managed instances by the productivity team inside engineering platform. It’s a lovely abstraction which they came up with, but simply put, people would ask for a new application to be created from our developer portal, choosing what tech stack to use and which environment. Whether they would want a private reverse proxy or a public one. And they would get their VM’s automatically created for the application along with the loadbalancers, along with mapping the DNS’s to the respective loadbalancers.</p>
<p>The VM’s would get lazily created i.e the first time any deployment would happen on them and then moving forward the records of the infrastructure, which got created would then map to the application for the evironments.</p>
<p>This was as compared to where developers would have to create all the Virtual machines via proctor manually and do the above by themselves. Which led to a lot of time saved while creating a new application for a team.</p>
<p>Another problem statement which got easier due to this VM creation API, was the process of deprecating virtual machines of certain language stacks/OS versions</p>
<p>Recreating VM fleets became easier and programmatic and people could now build their own tooling on how they wanted to create VM’s, which we ended up building for starters for deprecating VM fleets for managed services which we talked about earlier.</p>
<h2 id="why-didnt-we-add-it-in-proctor-itself">Why didn’t we add it in proctor itself?</h2>
<p>A couple of reasons. We already had a service which was used to create k8s clusters on demand. We wanted to add another resource to it, which would be used to create virtual machines, making it the central tool to orchestrate resource creation, the resource here could be anything, from a virtual machine to a k8s cluster.</p>
<p>We also felt it would be just cleaner as in interface, where proctor would be fronted by this orchestration service for resource creation and be the entrypoint for developers and the like for having any automation around resource creation needs with helper methods added to the client for their ease of use.</p>
<p>Could we have built everything back in proctor? Defintely. Would it work the same way as it does now? Mostly yes.</p>
<p>But we took to call to move the orchestration bit of every resource going forward to this new service. Be it orchestration of creation of new VM’s, new k8s cluster etc. This service would have the modelling to handle the creation, along with the inventory which we would get out the box (can be solved by other ways, proctor too in this case)</p>
<h2 id="moving-forward">Moving forward</h2>
<p>The end result was a lot of time saved in terms of developer time which they would have otherwise spent to create VM’s, cycle VM fleets with minimal developer intervention.</p>
<p>Which is the original and broader goal of automation too I feel, to move out more and more things out of the human hands so that people can move on to do other things. And as always, automation is a moving target for everyone, you build abstractions on top of abstractions sometimes to ease things out for everyone, without compromising on the usual things like quality of what is being delivered, ease of using.</p>
<h2 id="final-thoughts">Final thoughts</h2>
<p>Creating this along with <a href="https://twitter.com/vidit_m100">Vidit</a> and <a href="https://twitter.com/kartik7153">Kartik</a> was such a great experience. Thanks for the learnings.</p>
Handling language stack deprecations: Part 2: Container infrastructure
2021-06-15T00:00:00+00:00
https://www.tasdikrahman.com/2021/06/15/handling-language-stack-deprecations-part-2-container-infrastructure
<p>Given the number of language stacks which different product teams end up using inside the org, variations come, in the form of different versions being used, or different versions of dependent libraries coming in. This combination will quickly lead up to a whole set of container image variations for a particular language, crossed with operating system versions.</p>
<h2 id="what-is-the-problem-then">What is the problem then?</h2>
<p>Tracking what is being used by different product teams and their services arises when we would want to know what infrastructure combination is being used by different services. Tracking this piece of information is paramount for a couple of reasons for the central infrastructure team.</p>
<p>If you don’t know what people run, you wouldn’t know what to deprecate. Or in other words. You wouldn’t know what you support for the product teams.</p>
<h3 id="upgradesdeprecations">Upgrades/Deprecations</h3>
<p>For example, <a href="https://ubuntu.com/about/release-cycle">Ubuntu 16.04 LTS</a> recently got EOL’d, back in April 30th, 2021 where-in it stopped receiving security updates, creating an invetory of which services, in case they are using deprecated OS versions for example, would help in prioritizing and notifying the product teams about the impending upgrade.</p>
<p>This applies very well to the language stack deprecations needing to be done.</p>
<h3 id="library-additions">Library additions</h3>
<p>Catering to different requirements of library versions and variations of libraries required by each language+os stack variation also crops up. Deprecating/upgrading existing versions would need tracking as a pre-requisite.</p>
<h2 id="what-did-we-end-up-doing">What did we end up doing</h2>
<p>Given this would need to be solved for all applications inside the org, we started to tackle the problem starting with tracking the container images being used by the services onboarded to our internal service registry, to begin with and then handle the services which were not onboarded to the service registry.</p>
<center><img src="/content/images/2021/06/vesemir-deployment-flow.jpg" /></center>
<p>Given our deployment flow, the cli which we give out to the developers, is the first touchpoint in the CI/CD pipeline. The container images to be used for packaging and deploying the application are specified in the CI/CD pipeline configurations. Now we would want to parse this information present already by one of the scripts being used to either deploy/build/run test stage.</p>
<p>Given the simplicity of the solution, it would be simpler to just parse that specific string line for the container images for packaging the application and the image which is being used as the base image.</p>
<p>The deploy scripts are just a wrapper which make use of the cli interacting with the central service registry to deploy the application, along with added logic to handle the response</p>
<p>Next was to just add a simple route on the service registry which would recieve an HTTP request to receive this information, everytime a deployment would happen via the CI/CD UI. Leading to updation of this information for each application.</p>
<h2 id="current-state">Current state</h2>
<p>As of now this script of fetching this information is run for each and every k8s deployment which we run via our deployment automation.</p>
<p>This has in turn helped us create the inventory of all the language stacks which are actually being used by the
applications which are onboarded to our internal service registry and having deployments orchestrated via the same to our k8s clusters.</p>
<p>For deployments happening via other automation tools (helm for eg), we currently don’t have support to fetch what is the language stack being used.</p>
<h2 id="challenges-faced">Challenges faced</h2>
<ul>
<li>There were places where we wouldn’t be able to gauge the language stack version from the env vars which we would parse for an application, as there would be multiple versions present of that env var which were declared. We didn’t want to handle such cases initially, as we wanted to first cover the ideal cases first. Leaving this as an anamoly</li>
<li>Some applications hadn’t been deployed since months/years. Asking the devs to deploy was one solution, but there were cases where the services had no clear ownership, we didn’t want to deploy the application, to just fetch this information. We ended up creating an automation where we would fetch such applications from the source repository, running the parse logic and then making the http call to the service registry to store the information.</li>
</ul>
<p>The solution was not perfect, but this worked for us to build the initial language stack inventory for containers.</p>
<p>The next thing which we did was to plot this information on grafana to be easily accesible.</p>
<h2 id="future">Future</h2>
<p>We would want to then onboard our other applications which are not present in the service registry, to track their stack versions. Which would allow us to have more coverage on what are we currently supporting for the org, which would be the next step.</p>
<h2 id="takeaways">Takeaways</h2>
<ul>
<li>If we don’t know what we are running, we will not know what are we supporting</li>
<li>This becomes paramount, whenever we are trying to resolve security issues/add patches to software/support existing software</li>
<li>Not having this information would mean, getting surprised about the issues which come along with the versions of stacks which you are not aware of.</li>
<li>Piggybacking on existing workflows helped expedite the whole process of fetching the language stack information for an application</li>
</ul>
<p>If you liked what you read here, I have written a bit more about how we did the same for virtual machines <a href="https://www.tasdikrahman.com/2021/02/02/handling-language-stack-deprecations-part-1-virtual-machine-infrastructure/">here</a></p>
Revamping Vesemir: our virtual machine deployment service
2021-06-12T00:00:00+00:00
https://www.tasdikrahman.com/2021/06/12/revamping-our-virtual-machine-deployment-tooling
<p>This is a continuation of the <a href="https://www.tasdikrahman.com/2021/06/10/vesemir-our-virtual-machine-deployment-service/">post</a>, which details into the working of vesemir and how it goes about introducing changeset. Give it a read before continuing reading this, to allow you to gather more context on the what and the why.</p>
<p>While this post will focus more more on how we went on with revamping vesemir for increasing it’s reliability, maintainability and modernizing it.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Continuing the thread around how we did the same for Vesemir. (1/n) <a href="https://t.co/Xol0uRraJv">https://t.co/Xol0uRraJv</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1403746484630159360?ref_src=twsrc%5Etfw">June 12, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h3 id="previous-state-of-affairs">Previous state of affairs</h3>
<p>Trite as it is, everything in software I feel is improved over iterations and the same is true with vesemir here too. We have a functional piece of software which does things as expected and has been running for quite some time now. After the initial set of features, this project remained in a bug fix state for a bit.</p>
<p>When our team took ownership for the deployment ecosystem a year back, we started looking for ways in which we could better handle the support requests for deployment issues and to reduce the firefighting, given we were slowly starting to gain context bit by bit. The deployments for containers didn’t have much issues, given it was stable enough to not let such issues creep in.</p>
<p>For starters, there were a couple of boxes(VM’s) in which Vesemir was deployed, which would receive requests for deploying applications. We also noticed some teams independently cloning the initial vesemir VM’s, where they had added their own set of changes and then using this Vesemir VM to deploy their applications.</p>
<p>Centralized logging was something which we needed to add. Devs would look inside the specific VM boxes of vesemir to see what was going on, otherwise when a deployment failed (a very huge source of time sink for devs in our team/Level 1 support folks).</p>
<p>Monitoring was missing on the boxes, if and when CPU would spike (continuing to stay that way), we would not know about it and the deployments being processed would get slower and take more time. This would cause deployment failures, a bit of that is explained here in this <a href="https://www.tasdikrahman.com/2021/06/06/bug-which-would-cause-some-deployments-to-get-triggered-again-and-again/">post</a>, as the workers picking up deployment requests were having a finite time to process the deployment request.</p>
<p>Integration testing for the changes before deploying them to the production set of vesemir VM’s was very manual and haphazard, prone to manual errors. It involved, removing one of the vesemir boxes, from behind the HAproxy and then deploying the changes to this vesemir box, testing it all along, and if everything went well, doing the same for other boxes. Again a huge time sink.</p>
<p>The way we deployed vesemir to the vesemir VM was very manual, someone would have to remove the box from the HAproxy manually, pull the changes to the boxes, and then restart the service, being supervised by systemd. The whole process of deploying would take around ~10-15mins for the whole set of vesemir boxes.</p>
<p>Before starting with anything, we would want to solve these issues for basic sanity of vesemir.</p>
<h3 id="what-did-we-fix-first">What did we fix first</h3>
<p>As with everything, prioritization is crucial. Breaking down the tasks into tactical and strategic fixes so as to fix what was burning.</p>
<p>Fixes which would immediately give us value and would help us reduce, the firefighting were prioritised first. Which would help us get the breathing space.</p>
<p>Fixes/features which would help us long term value over a larger segment of users, re-architecting the setup, providing better reliability, refactoring which would help us maintain the codebase better were kept for the end.</p>
<h3 id="baseline-fixes">Baseline fixes</h3>
<p>To solve the onboarding issue of other devs in the team and to reduce the backlog of support requests received by our team for deployment failures, we ended up creating a playbook which would consist of the different failure modes for vesemir and their solutions. Which we gave to our platform support team (more on this model later), who would then look at the tickets and then solve the customer deployment failure issues. If there would be something which they wouldn’t be able to help with, we would come in as the second line of defence. This helped us immensely in freeing up time to solve the initial burning issues which we were seeing with vesemir.</p>
<p>Our team ended up picking, adding logging support to Vesemir, which would be then be pushed to a centralized logging platform. This would allow us, the support team, as well as the developers of the product teams to look into what really caused the issue and then give us that initial bit of information about what went wrong for further debugging.</p>
<p>Along with it, we also consolidated the vesemir VM’s and put them behind an HAproxy box, to front the service and loadbalance the incoming requests, which was missing earlier.</p>
<p>We also picked up adding monitoring and alerting for the vesemir VM’s which would allow us visibility into what was happening. This also allowed us to see the CPU going all the way upto 100%, without any signifant increase in deployment requests being received.</p>
<p><a href="https://twitter.com/viditganpi/">Vidit</a> has written a <a href="https://www.gojek.io/blog/how-we-reduced-skyrocketing-cpu-usage">great article</a> on solving for the same , where we ended up going from python2 to python3, which helped fix the issue altogether of the CPU spiking and staying that way, helping us reduce the number of VM’s which we had been using.</p>
<p>We ended up replicating the production setup for an integration setup, which allowed us to have an exact replica of the production setup. The integration environment of the deployment orchestrator(also our service registry), was updated to start pointing to the integration environment of vesemir which we just created. This helped us test end to end, by sending deploy requests from the cli which we give to the developers. Allowing us to catch the errors before them landing up on production.</p>
<p>There were a couple of major bug fixes which we added, helping decrease the defect rate and increasing the reliability of deployments going through.</p>
<p>Deleting dead code, adding missing tests, a linter, integration testing scenarios, along with refactoring the codebase next, helped us maintain the codebase better and helping new devs getting onboarded to it much faster then before.</p>
<p>Adding support for storing deployment metadata like which application is getting deployed, the number of VM’s which we are deploying, the time it’s taking for the deployment, the chef tags being used to search for the application etc were also a few things which we started persisting in the vesemir database, helping us track metrics for deployments.</p>
<h3 id="deploying-vesemir-by-the-service-which-is-used-for-deploying-to-kubernetes">Deploying vesemir by the service which is used for deploying to kubernetes</h3>
<p>The next big chunk of work which we wanted to tackle was to automate the manual deployment process of vesemir, which would not only be time consuming for a developer in our team but would also be error prone whenever someone would be doing it.</p>
<p>This is where normandy came into picture. More on normandy in another post, but it’s a service written on top of go-client which would effectively be used for doing CRUD operations on kubernetes resources required by an application when it is getting deployed to a kubernetes cluster(s)</p>
<p>The immediate benefit coming out of deploying vesemir via normandy, was that now what used to take more than 10-15mins of manually deploying vesemir to the VM’s. We now were having a system in place where we would be able to deploy vesemir with the help of our deployment tooling which we give out to the product developers, with the same deployment UX, along with rollback being available via the same interface. Helping us dogfeed our own service to us.</p>
<p>We ended up adding support for python stack based deployments via our deployment platform as part of this effort, which came as a strategic win for us.</p>
<p>Another big win for us, was that we now knew the exact dependencies for vesemir, on what it used and what it didn’t, which made it’s infrastructure reproducible and immutable.</p>
<h3 id="outcome">Outcome</h3>
<p>The biggest outcome of this revamp, came in terms of reducing the defect rate by ~18% from what it was earlier.</p>
<p>Monitoring, logging and reliability changes, along with better testing mechanisms introduced, helped reduce the hesistation by devs diving deeper into vesemir’s codebase, which helped in taking it from a codebase which people feared to introduce changes into to something where the changeset could now be introduced in a couple of minutes, with immutability baked in.</p>
<p>A lot of toil involved in maintaining/adding features/testing to vesemir was reduced, which helped with developer productivity.</p>
<p>We upgraded from an EOL’d version of python, which no longer was receiving any security fixes.</p>
<h3 id="why-didnt-we-just-rewrite-vesemir">Why didn’t we just rewrite vesemir?</h3>
<p>There were a couple of things, which we considered before deciding upon not to rewrite vesemir.</p>
<h4 id="cons-of-a-rewrite">Con’s of a rewrite</h4>
<ul>
<li>the new piece will be missing the vesemir codebase which has been
<ul>
<li>well tested for the behaviour by end-users.</li>
<li>in used for a couple of years.</li>
<li>patched with a lot of bug fixes which have been found and fixed over time with usage.</li>
</ul>
</li>
<li>not taking into consideration the dev effort which will be required to maintain two systems during such a migration to the new tool, where-in making us maitain two systems at the same time.</li>
</ul>
<h4 id="pros-of-a-rewrite">Pro’s of a rewrite</h4>
<ul>
<li>Completely new codebase, following basic sanity checks and best practices, right from the start.</li>
<li>Rewrite in a language more familiar with the rest of our teams codebases(majority being in ruby/golang) rather than being the single service written in python.</li>
<li>Not necessary, that the shortcomings solved in this rewrite would overshadow the new bugs which would come in as part of the rewrite.</li>
</ul>
<h3 id="takeaways">Takeaways</h3>
<p>Learned a bunch during the course of the whole revamp, while creating the project plan and cutting out user stories for this, but the biggest for me was learning prioritization, deciding on what is more important of a problem to solve which would create more impact, helping the end users achieve(the product developers) achieve the end result of introducing their changeset for their service, in a reliable manner.</p>
<p>This was also the time when fresh grads had joined our team, mentoring them to ramp up on the codebase and the operational side of things was something which helped me deepen my own understanding around our domain.</p>
<p>This also was a good experience in dealing with a really hairy, legacy piece of software, working around it while enhancing it at the same time without causing issues to the customers(product team devs) and was something new.</p>
<p>Thanks for reading this piece. More on around normandy, our kubernetes deployment service, the other part of our deployment platform in another post.</p>
<h3 id="links">Links</h3>
<ul>
<li><a href="https://www.gojek.io/blog/how-we-reduced-skyrocketing-cpu-usage">https://www.gojek.io/blog/how-we-reduced-skyrocketing-cpu-usage</a></li>
<li><a href="https://www.tasdikrahman.com/2021/06/10/vesemir-our-virtual-machine-deployment-service/">https://www.tasdikrahman.com/2021/06/10/vesemir-our-virtual-machine-deployment-service/</a></li>
<li><a href="https://unsplash.com/photos/8lvHMctQMrU">Cover image credits</a></li>
</ul>
Vesemir: Our virtual machine deployment service
2021-06-10T00:00:00+00:00
https://www.tasdikrahman.com/2021/06/10/vesemir-our-virtual-machine-deployment-service
<p>This post is a continuation of the tweet here</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">The build and deployment pipeline for each org will be different in some way or the other, given each co will have it’s own requirements. This thread talks a bit about our virtual machine deployment pipeline (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1403038789052735491?ref_src=twsrc%5Etfw">June 10, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>This was also cross posted originally for the gojek tech blog https://www.gojek.io/blog/introducing-vesemir-gojeks-virtual-machine-deployment-service</p>
<p>The build and deployment pipeline for each org will be different in some way or the other, given each co will have it’s own requirements. Even in my last org, the way our team enabled other teams to ship code/config changes, was pretty different from the way we do it in my current org.</p>
<h2 id="background">Background</h2>
<p>Our deployment platform comprises, of a central service registry, two separate services to deploy to virtual machines and kubernetes respectively, a cli to interact with the service registry, which is given out to the developers of the org using which they can do numerous things for their application in a self serve manner. One of those things, being the ability to deploy their applications via the CLI.</p>
<p>The idea is to abstract out the deployment process from the developers as much as possible, but also giving them the means to debug with the logs being shown to them, which are meaningful to them to figure out what went wrong if it doesn’t work as expected.</p>
<p>For this post, I will specifically talk about the virtual machine deployment service, called vesemir, which we have in place and how we went about revamping it’s infrastructure along with refactoring it to keep it healthy and maintainable for others to work on it’s codebase.</p>
<p>Given, GCP doesn’t have a service similar to <a href="https://aws.amazon.com/codedeploy/">AWS codedeploy</a>, vesemir came out of the same requirement.</p>
<h3 id="what-does-it-really-do">What does it really do?</h3>
<p>Vesemir is a inhouse python service, which in a gist is a wrapper on top of chef API’s, where-in it receives a request for deploying a service in a particular cluster (a GCP project), filters out the VM’s where it has to deploy the changeset and then goes about deploying the changeset, either one at a time or at the level of concurrency insisted upon by the request.</p>
<p>It hels us deploy to integration and production environments, 300+ times every day.</p>
<h3 id="how-does-it-do-it">How does it do it?</h3>
<p>The initial request payload, bits which are of most relevance are</p>
<ul>
<li>application name</li>
<li>environment</li>
<li>team</li>
<li>chef tags to be used for filtering the application VM’s</li>
<li>haproxy metadata (degradation time specified, cookbook/recipe/tags to filter the haproxy boxes, haproxy backend)</li>
<li>concurrency (number of VM’s to deploy at a time)</li>
</ul>
<p>Once vesemir, receives this piece of information, it queries the chef server via <a href="https://github.com/coderanger/pychef">pychef</a>, with the information passed for the search tags, the query for which gets constructed and send to the central chef server. The hostnames and their IP’s then get parsed.</p>
<p>This takes care of the first bit of the problem where we know which VM’s are to be touched for deploying the changeset.</p>
<p>Along with this bit, the <a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/executor/playbook_executor.py#L41">playbook.PlaybookExecutor</a> gets initialised with the ansible playbook(more on this later) which we would be executing on the hosts found, the inventory file, and the variable manager named argument taking in extra variables which are going to be used by the templatised playbook.</p>
<p>We write the ansible inventory file in a temporary file, using <a href="https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile">NamedTemporaryFile</a> which gets deleted after the whole request/response lifecyle for a deployment, which gets fed into the <a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/inventory/manager.py#L118">InventoryManager</a>, where we pass the hostnamefile which we created above as <a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/inventory/manager.py#L137">sources</a></p>
<p>The ansible playbook which gets fed into <code class="language-plaintext highlighter-rouge">playbook.PlaybookExecutor</code>, is a static playbook with jinja templating, where we feed the options via the <a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/vars/manager.py#L76">VariableManager</a> into the <a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/vars/manager.py#L85"><code class="language-plaintext highlighter-rouge">extra_vars</code></a></p>
<p>The options named var, will take in details like, which user and other authz/authn details to be used by ansible to ssh into the hosts, along with options like <code class="language-plaintext highlighter-rouge">forks</code> where the concurrency is set for the number of boxes where we would want to run the playbook.</p>
<p>The next step is to execute the <a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/executor/task_executor.py#L84">run</a> method on the initialiazed object of <code class="language-plaintext highlighter-rouge">playbook.PlaybookExecutor</code>. The important bit here after the run is the collection of stats which we pick up, from the <a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/executor/task_queue_manager.py#L52">TaskQueueManager</a>, which is what we then use to check for nodes where the playbook ran successfully or not, if the hosts were unreachable and so on.</p>
<p>This piece of information is then used to form a response to be given back to the client which has called vesemir.</p>
<p>As for the playbook and what does it do, in gist, it first, disables the server where it is first going to deploy, in the haproxy backend for the application servers. Introduces the changeset, restarts the service, enables this VM back in the HAproxy backend with weights if provided during the request, an option to sleep for a bit is also introduced which is again controlled by the request, before which the VM is inserted back with 100% weight.</p>
<p>The playbook is then looped over the hosts, returned by the chef query while searching, all while executing the playbook tasks on the hosts.</p>
<center><img src="/content/images/2021/06/vesemir-deployment-flow.jpg" /></center>
<h2 id="pros-of-this-deployment-pattern">Pros of this deployment pattern</h2>
<p>The obvious is since we control the deployment platform tooling, it allows for us to have a level of flexibility, not possible with vendors. Everything is more or less, an outcome of this flexibility which we get.</p>
<h2 id="shortcomings-of-this-deployment-pattern">Shortcomings of this deployment pattern</h2>
<p>Given the changelog is introduced in each VM in such a way, immutability is not possible, which is again to the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">AMI</a> based approach, where AWS codedeploy would do something similar to inject the codebase into the target VMs.</p>
<p>More moving parts in the system, which depend on external dependencies. For eg: ansible being a very important part for vesemir to deploy the changeset, right from using piggybackin on the ssh interface provided, to the templatising of the steps of execution to be done.</p>
<p>Maintenance of a custom tool, which would have it’s own lifecycle, maintenance, bugs and feature requests.</p>
<h2 id="takeaways">Takeaways</h2>
<p>There are numerous ways in which people end up deploying their codebases, but maintaining vesemir for sometime now for my current org, having seen both sides of having immutable deployments via container images, to deploying changeset via vesemir, it’s not very hard to see the selling point of immutability.</p>
<p>Thanks for reading this piece so far, more on how we ended up revamping vesemir in another post.</p>
<h3 id="links">Links</h3>
<ul>
<li><a href="https://github.com/ansible/ansible/blob/v2.5.1/lib/ansible/executor/playbook_executor.py">playbook.PlaybookExecutor class - https://github.com/ansible/ansible/blob/v2.5.1/lib/ansible/executor/playbook_executor.py</a></li>
<li><a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/vars/manager.py#L76">VariableManager class - https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/vars/manager.py#L76</a></li>
<li><a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/executor/task_executor.py#L84">TaskExecutor class - https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/executor/task_executor.py#L84</a></li>
<li><a href="https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/executor/task_queue_manager.py#L52">TaskQueueManager class - https://github.com/ansible/ansible/blob/v2.5.0/lib/ansible/executor/task_queue_manager.py#L52</a></li>
<li><a href="https://github.com/coderanger/pychef">https://github.com/coderanger/pychef</a></li>
<li><a href="https://unsplash.com/photos/ejqfevj3Xv8">Cover image credits</a></li>
</ul>
Bug which would cause some deployments to get triggered again and again
2021-06-06T00:00:00+00:00
https://www.tasdikrahman.com/2021/06/06/bug-which-would-cause-some-deployments-to-get-triggered-again-and-again
<p>This post is a continuation of this tweet here.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">We recently encountered a bug in our deployment flow, which we were completely oblivious to. (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1401225434835001345?ref_src=twsrc%5Etfw">June 5, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Bugs are present in every system, waiting to be discovered. As such, this one was no different.</p>
<h2 id="what-did-the-bug-do">What did the bug do?</h2>
<p>Would cause an application to be deployed again and again, even though when it had been triggered only once to be deployed.</p>
<h2 id="context-about-the-system-involved">Context about the system involved</h2>
<p>The part which handles requests to deploy an application to a set of VM’s is a custom service which we have written and maintained over the years. The exact workings of it is the subject for another discussion. But gist is that, it has an API which listens to requests and does the intended work of deploying a change set to a set of VM’s for the application and would return back an appropriate response.</p>
<p>Depending on whether the deployment was successful or not, the response body gets added with further information for the same for the client here to decipher whether the deployment was a success or not.</p>
<p>The deployment service runs behind gunicorn with a couple of sync workers handling the incoming requests, only other thing being that the workers are configured with a <a href="https://github.com/benoitc/gunicorn/issues/1493#issuecomment-321331753">timeout configuration</a>, which acts as request timeout configuration.</p>
<p>The current setup being sync, there is a timeout configuration also present on the reverse proxy which sits between the client and the service which deploys an application to VM’s. The proxy timeout configuration was set to the same value as the gunicorn worker timeout.</p>
<p>Orchestration for the whole deployment, right from sending the details of the deployment request (which app to deploy, which environment along with other metadata), is handled by a separate service, which also doubles up as the internal service registry (more on this later)</p>
<p>The deploy request which this broker service receives, eventually ends up being queued as a background job. One detail here being that the job processing framework used here, retries failures with an exponential backoff by default.</p>
<p>Which would mean that the retry will keep on happening for a couple of times, which ends up being a not so good number for the nature of the job beind handled, which is not an intended behavior for this flow. After the max no’s retries the job would be pushed to the deadset</p>
<h2 id="immediate-fix">Immediate fix</h2>
<p>After killing this specific job manually, to immediately resolve the issue, we went a bit further into checking the cause because of which the job processing logic for the deployment job was causing an error, leading to the retry.</p>
<h2 id="what-caused-it">What caused it</h2>
<p>Now the way the service handling deployments works is such that, out of a set of application VM’s, where it has to deploy, it will do it either 1 at a time or the level of concurrency being passed in the deployment request.</p>
<p>More on the workings of the series of steps taken while deploying this changeset later. The point to note here being that the deployment time increases as a factor of the number of VM’s present for the application, in the particular environment where it is being deployed.</p>
<p>What ended up happening was, the orchestrator would wait for the response from the deployment service, after x minutes set on the gunicorn worker, set same as the reverse proxy connection timeout too, the request would be abruptly closed by the proxy if it exceeded x mins.</p>
<p>All of this, even before a valid response could be received by the orchestrator from the deployment service.</p>
<p>The block handling the response from the deployment service, was having a guard to handle an HTTP error while making the request, but it was after the block where we would check for the status of a deployment after the response was received.</p>
<p>Given the timeout configuration, the HTTP error flow not being handled before the parsing the deployment response flow, it would end up error-ing out the deployment job, leading it to be retried by the background job processor.</p>
<h2 id="what-did-we-do-to-prevent-this-from-happening-again">What did we do to prevent this from happening again</h2>
<p>The first fix added was to increase the response timeout in the reverse proxy to be comfortably more than the timeout set on the gunicorn timeout, so that if the worker times out, the response would be sent through, without the connection getting timed out by the proxy.</p>
<p>The second thing which we ended up doing was to keep a check on the deployment flow, before initiating the deployment on whether the deployment is already in a terminal state (failed/succeeded) which was easy to check given the state transitions being maintained.</p>
<p>We could have also made use of the background job processor’s <code class="language-plaintext highlighter-rouge">max_retry</code> setting in this case by setting it to 0, which would have not retried the job at all if it failed once. Another option would have been to use <code class="language-plaintext highlighter-rouge">discard_on</code> here <a href="https://edgeapi.rubyonrails.org/classes/ActiveJob/Exceptions/ClassMethods.html#method-i-discard_on">https://edgeapi.rubyonrails.org/classes/ActiveJob/Exceptions/ClassMethods.html#method-i-discard_on</a></p>
<p>Although the root cause is not this, but an obligatory plug, of the fallacy of <a href="https://web.archive.org/web/20171107014323/http://blog.fogcreek.com/eight-fallacies-of-distributed-computing-tech-talk/">“Network is reliable”</a></p>
<h2 id="links">Links</h2>
<ul>
<li><a href="https://github.com/benoitc/gunicorn/issues/1493#issuecomment-321331753">https://github.com/benoitc/gunicorn/issues/1493#issuecomment-321331753</a></li>
<li><a href="https://docs.gunicorn.org/en/stable/settings.html#timeout">https://docs.gunicorn.org/en/stable/settings.html#timeout</a></li>
<li><a href="https://unsplash.com/photos/ZmFjEJq-x9k">Credits for the cover photo to Hitanshu</a></li>
</ul>
Handling language stack deprecations: Part 1: Virtual Machine infrastructure
2021-02-02T00:00:00+00:00
https://www.tasdikrahman.com/2021/02/02/handling-language-stack-deprecations-part-1-virtual-machine-infrastructure
<p>This post is a continuation of this <a href="https://twitter.com/tasdikrahman/status/1355129674192490498?s=20">tweet thread</a></p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Language deprecation for stacks can be a task if you are on VMs, added to that the confusion on what version of that stack runs, in your inventory if it's not small. Summarizing what we ended up doing to bring visibility & giving people the ability to migrate themselves (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1355129674192490498?ref_src=twsrc%5Etfw">January 29, 2021</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="compute-vms">Compute VM’s</h2>
<p>Given the nature of VMs and how they are run and created in our compute infrastructure environment. Managing, upgrading and adding fixes to them becomes a task in itself. Given that there is no control plane to control the lifecycle of these VM’s, the task is manual at best even though there is automation to delete and create VMs on demand (more on the VM creation API which we created in a different post).</p>
<p>If the workloads were on kubernetes, the deprecation step is as simple as a simple deploy of the application, after the base images would have been updated with the necessary changes required to deprecate the language stack.</p>
<p>Given the confusion on which language stack version ran on which VM’s of our inventory, this only added to the problem statement. Which is clear in itself, that we have to deprecate a group of VM’s which are running a particular language stack without causing any disruption to the workloads running on them.</p>
<h2 id="where-to-start">Where to start?</h2>
<p>To begin with, the automation would need fixing to disallow the creation of the compute infrastructure with the language stack which is out for deprecation. This would help remove the moving target of VMs which would get created with the automation you have. Having a finite number helps reduce the effort at the end.</p>
<h2 id="creating-the-inventory">Creating the inventory</h2>
<p>Along with that, the next natural step would be the creation of the inventory of which set of VM’s are running the version of language stack which you would want to deprecate.</p>
<p>If the number of VM’s are in a few hundreds combined for the org, one can simply check their <a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">IAC</a> configuration which they would have checked into their git repo’s, which they had applied to create the infrastructure.</p>
<p>Manual for an effort, but for starters would work just fine due to the smaller number of VM’s.</p>
<p>In contrast, if the inventory of VM’s is running in thousands of VM’s, the same approach is impractical at best. Furthermore, the whole idea of maintaining the IAC for these set of VM’s is counter-intuitive given the amount of config which would have to be maintained.</p>
<p>Compute at present is created on demand. A dev needing a compute VM can simply go ahead and use proctor (our in house automation orchestrator) to create a VM for them. (More on this in this <a href="https://www.youtube.com/watch?v=mE1JZKMhnNs">talk</a></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/mE1JZKMhnNs" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>But what about inventory? While creating the VM, we add tags (both on the configuration management tool, along with the cloud provider resource which is created)</p>
<p>This allows someone to query the VMs with regards to which team/group the VM belongs to. While creating the VMs if your current automation is missing the addition of the language version stack version which it is going to create the version with. Which also makes the whole job very simple in terms of just querying your configuration management tool to get your VM’s for the query you would like to give it. In our case, we didn’t have these tags for the language stack versions.</p>
<p>Which brings us to the situation, where it’s not clear on which VM’s are actually running which version of a language stack. As to the number of language stack versions running in the VM’s would vary as and when support for different versions would have been added over time.</p>
<p>How did we go about it? Given the number of VM’s, it was best to have this generated in an automated manner, simply because of the impracticality of this being tackled in a manner which involved someone checking this and creating it manually. Along with the fact that, the VM infrastructure would keep increasing over time, the data would go stale really quick.</p>
<p>As this was also something which would have to be repeated for another language stack/os version/etc again sometime in the future, it was best to have some sort of a repeatable and predictable process which could be leveraged to do this in a faster manner.</p>
<p>The solution as of now is an airflow job, which runs at a specific time frame, after the execution of which, the end result are a few artifacts which it creates/updates</p>
<p>The artifact in our case being, a google sheet which will have entries of all VM’s which are running the particular language stack, along with metadata for eg: ownership details, os version, production/integration, what kind of compute VM is it etc.</p>
<p>What the airflow job does, apart from this, is to keep updating a RDBMS with information similar to what it inserted in the google sheet.</p>
<p>The RDBMS got imported as a data source on grafana, on top of which we gave the end users the VM’s under their ownership for that particular language stack. This allowed them to have a simple interface to the VM set, without the need for them to know any internal details.</p>
<h2 id="how-does-having-different-view-layers-for-this-inventory-help">How does having different view layers for this inventory help?</h2>
<p>The dev folks would simply look at the VM hostname/IP and replace it with the automation they are familiar with to create a new VM to be used in place with older VM, giving them the power to do the whole activity without getting blocked on anyone</p>
<p>To keep track of how many VM’s each product group/team’s progress, what we also ended up doing was, keep a track of the number of VM’s they had for that particular language stack at the end of each week.</p>
<p>This number, would be then sent to the teams/product groups as an email which is powered by another scheduled airflow job, along with a grafana dashboard for viewing the set of instance and docs required for doing the whole activity</p>
<p>The following dashboard for example, shows the trend line for the number of VM’s on a particular language stack and their numbers over time for different teams/product groups</p>
<center><img src="/content/images/2021/02/vm-inventory-trendline.jpg" /></center>
<h2 id="more-on-the-automation-used">More on the automation used</h2>
<p>Coming back to the airflow automation job specifics, the job which creates/updates the google sheet and inserts VM details to the RDBMS.</p>
<p>It makes use of the configuration management tool’s language specific client which is used in the script, to first filter the VM’s based on a specific automation template which gets attached to the VM when it gets registered to the configuration management tool.</p>
<p>This helps first sort the VMs which would be running some version of the language stack. What the script does next is, ssh into these VM list in batch and pluck out the details of the language stack by running a command which would be specific to the language stack</p>
<p>Which would then blurt out the version information, the script captures this information from the VM and logs out. A few other helper commands are run as part of the whole script, which would pluck out the metadata relevant and create/update the artifacts necessary.</p>
<p>The helper library created here, is flexible enough to take input in the form of the query which you would want to run on the VM to capture the relevant information, which helps in future inventory management.</p>
<p>Would love to hear what you folks do for this repetitive chore and how you do it.</p>
<p>If you liked what you read here, I have written a bit more about how we did the same for containers <a href="https://www.tasdikrahman.com/2021/02/02/handling-language-stack-deprecations-part-1-virtual-machine-infrastructure/">here</a></p>
Maintaing aptly - The debian package manager
2020-12-23T00:00:00+00:00
https://www.tasdikrahman.com/2020/12/23/maintaining-aptly-the-debian-package-manager
<p>This post is a continuation of this <a href="https://twitter.com/tasdikrahman/status/1326536874375090176">tweet thread</a></p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Sad to see aptly slowly <a href="https://t.co/zkXgsruAGi">https://t.co/zkXgsruAGi</a> rotting, but works really well till the last 1.4.0 build as a debian package repository for your needs. (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1326536874375090176?ref_src=twsrc%5Etfw">November 11, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p><a href="https://www.aptly.info/">Aptly</a> is a debian package repository, the specific use case which we are using it for is pushing out application specific debian packages which will then be pulled out while deploying a new SHA/version of the application, to the app boxes. More on this in another post. But what this post will concentrate on, are a few things which we discovered while maintaining aptly, storing packages which ran into storage spaces consuming multiple TBs.</p>
<h2 id="use-an-ssd">Use an SSD</h2>
<p>This is a must here, since aptly serves the debian packages straight from the filesystem. If you are not using an SSD, you will definitely see a slowup in the package fetch/insert steps.</p>
<h2 id="add-your-cleanup-scripts-early">Add your cleanup scripts early</h2>
<p>Make it known the application owners, that you will only keep, say last 10-15 packages in the filesystem. Given that the deb package would roughly take a few hundred MiBs(very rough estimate, can vary for you) you will reach a point where your initial storage will run out over time.</p>
<p>Throwing disk and compute can be done and is all fine, but the former will reach a limit, when your filesystem itself can’t support it any further (ext4, supports logically uptil 1Exbibyte (1EiB), but the point is that you don’t really end up in a situation like that where cleanup becomes a problem)</p>
<p>Even then, resize2fs (version 1.42.9) would simply fail if you tried increasing the disc more than 16TB, saying that the new size is too large to be expressed in 32bits. Adding to it, if you are running an older kernel version, eg 3.19.x which doesn’t handle 64 bit ext4 filesystems properly, you would be left in a puddle here.</p>
<h2 id="build-time-slows-down-over-time">Build time slows down over time</h2>
<p>As with time, the index of the packages held inside aptly will grow, which will considerably increase the package publish step for your package consumers, which in most environments is a huge productivity kill. Imagine having to wait x amounts of minutes every now and then while trying to push a commit and deploying it over to the app boxes. If you combine this to the number of developers in the team/company, those are a lot of people hours right there.</p>
<p>The particular step is the package publish step, for a distribution which slows the whole process.</p>
<h2 id="how-to-prevent-this-api-slowdown-in-the-publish-step">How to prevent this API slowdown in the publish step?</h2>
<p>The publish step <a href="https://www.aptly.info/doc/aptly/publish/repo/">https://www.aptly.info/doc/aptly/publish/repo/</a> has an option/flag called <code class="language-plaintext highlighter-rouge">--skip-contents</code>, which will essentially not generate the index of the contents stored. We had tried unsuccessfully storing this in the aptly app config, but it seemed to not work.</p>
<p>After checking the <a href="https://github.com/aptly-dev/aptly/blob/24a027194ea8818307083396edb76565f41acc92/api/publish.go#L232">codebase</a>, in the specific route, which was <code class="language-plaintext highlighter-rouge">/publish/:prefix/:distribution</code> which used to take the most time, and for which we wanted to set the above setting.</p>
<div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// https://github.com/aptly-dev/aptly/blob/24a027194ea8818307083396edb76565f41acc92/api/publish.go#L232</span>
<span class="c">// PUT /publish/:prefix/:distribution</span>
<span class="k">func</span> <span class="n">apiPublishUpdateSwitch</span><span class="p">(</span><span class="n">c</span> <span class="o">*</span><span class="n">gin</span><span class="o">.</span><span class="n">Context</span><span class="p">)</span> <span class="p">{</span>
<span class="n">param</span> <span class="o">:=</span> <span class="n">parseEscapedPath</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">Params</span><span class="o">.</span><span class="n">ByName</span><span class="p">(</span><span class="s">"prefix"</span><span class="p">))</span>
<span class="n">storage</span><span class="p">,</span> <span class="n">prefix</span> <span class="o">:=</span> <span class="n">deb</span><span class="o">.</span><span class="n">ParsePrefix</span><span class="p">(</span><span class="n">param</span><span class="p">)</span>
<span class="n">distribution</span> <span class="o">:=</span> <span class="n">c</span><span class="o">.</span><span class="n">Params</span><span class="o">.</span><span class="n">ByName</span><span class="p">(</span><span class="s">"distribution"</span><span class="p">)</span>
<span class="k">var</span> <span class="n">b</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">ForceOverwrite</span> <span class="kt">bool</span>
<span class="n">Signing</span> <span class="n">SigningOptions</span>
<span class="n">SkipContents</span> <span class="o">*</span><span class="kt">bool</span>
<span class="n">SkipCleanup</span> <span class="o">*</span><span class="kt">bool</span>
<span class="n">Snapshots</span> <span class="p">[]</span><span class="k">struct</span> <span class="p">{</span>
<span class="n">Component</span> <span class="kt">string</span> <span class="s">`binding:"required"`</span>
<span class="n">Name</span> <span class="kt">string</span> <span class="s">`binding:"required"`</span>
<span class="p">}</span>
<span class="n">AcquireByHash</span> <span class="o">*</span><span class="kt">bool</span>
<span class="p">}</span>
<span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">Bind</span><span class="p">(</span><span class="o">&</span><span class="n">b</span><span class="p">)</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="o">...</span>
<span class="o">...</span>
</code></pre></div></div>
<p>The var <code class="language-plaintext highlighter-rouge">b</code> would get it’s de-serialised content for <code class="language-plaintext highlighter-rouge">SkipContents</code> is what we anticipated, when we started passing the value <code class="language-plaintext highlighter-rouge">skip-contents: true</code>(as we saw in the <a href="https://www.aptly.info/doc/aptly/publish/repo/">docs</a>), in the <code class="language-plaintext highlighter-rouge">PUT</code> call, as part of the final package step.But this also seemed to not work.</p>
<p>After digging a bit more, <a href="https://twitter.com/viditganpi/">Vidit</a> and <a href="https://twitter.com/kartik7153/">Kartik</a> discovered that we were passing the wrong header key which would be used to Skip contents, from their test suite, the configuration to allow <a href="https://github.com/aptly-dev/aptly/blob/24a027194ea8818307083396edb76565f41acc92/system/t12_api/publish.py#L49">setting this value</a>, the header to be passed was <code class="language-plaintext highlighter-rouge">SkipContents: true</code>, after making the change. This was the step which was taking all the time as the package index size grew over time, making the whole package upload step slow.</p>
<h2 id="alternative-ways-of-tackling-the-package-index-step-slowdown">Alternative ways of tackling the package index step slowdown</h2>
<p>One route, which can be taken would be, to create a new aptly instance, start pushing your debian package files to this instance instead rather than the older slower one.</p>
<p>How would the client pick up packages from both these apt sources? Multiple sources can be specified in files under <code class="language-plaintext highlighter-rouge">/etc/apt/sources.list.d/</code>, which will be looked up while searching for a package. For newer packages, the client will pick it up from the newer apt source (new aptly instance) and for the older ones, it will pick it from the older apt source (older aptly instance)</p>
<p>Will immediately solve the problem of slower builds, as well as the sideeffect of having a clean newer setup, where you can start enforcing the standard of keeping foo number of packages from now on</p>
<p>The other solution is to either self host another alternative like Pulp3 etc, or a paid package manager, which would take away some bits of these off your plate.</p>
<p>Another thing to note here is that, aptly hasn’t had a commit on it’s master for quite some time along with a new release not being put our for some time now. There has been an open <a href="https://github.com/aptly-dev/aptly/issues/920">issue</a> regarding the question of whether it’s maintained anymore. Although, I personally feel it’s feature complete for the set of features we have been currently using, and runs without any fuss whatsover for the most part, this is definitely something which you should consider as something while weighing down on options.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://www.aptly.info/doc/aptly/publish/repo/">https://www.aptly.info/doc/aptly/publish/repo/</a></li>
<li><a href="https://github.com/aptly-dev/aptly/blob/24a027194ea8818307083396edb76565f41acc92/system/t12_api/publish.py#L49">https://github.com/aptly-dev/aptly/blob/24a027194ea8818307083396edb76565f41acc92/system/t12_api/publish.py#L49</a></li>
<li><a href="https://github.com/aptly-dev/aptly/issues/920">https://github.com/aptly-dev/aptly/issues/920</a></li>
</ul>
<h2 id="credits">Credits</h2>
<ul>
<li>Post header image credits: www.aptly.info</li>
</ul>
What to avoid while doing PR reviews
2020-12-14T00:00:00+00:00
https://www.tasdikrahman.com/2020/12/14/what-to-avoid-while-doing-PR-reviews
<p>As with time this doc will change, but jotting my thoughts down here on what I feel I would like to avoid while I review PR’s.</p>
<h2 id="code-formattingstyle-suggestions">Code formatting/style suggestions</h2>
<p>I believe it’s best left to the machine to do this instead of a human trying to fixate their attention to this, given it takes away the precious time of the reviewer which could be diverted to review the crux of the changes which the submission tries introducing. A code formatter should ideally pick this step up from the human reviewers’s plate. An opinionated code-formatter/linter/style checker is the best option to have. An example for this will be <a href="https://golang.org/cmd/gofmt/">gofmt</a>/linter which weeds out code formatting issues right in the build/test step. <a href="https://github.com/rubocop-hq/rubocop">rubocop</a> is another great example.</p>
<p>What a tool is not able to enforce should be added as a team guide when reviewing PR’s, opining about certain styles and blocking the merging of the PR is not the best way to deal about changes being introduced, worse yet. If the person is new/not someone from the team, would only add up to the friction of contributing towards the codebase, which is not a good sign.</p>
<p>If there is something which the reviewer is very concerned as a style in the PR, they should ideally leave it be for this changeset if it was not caught by the automated tooling/not mentioned in the team style guide and raise it with the team to be voted upon from the next PR’s.</p>
<p>Obvious thing to avoid here is going against the language guidelines themselves, I would preferably rather stick to the language guidelines(<a href="https://www.python.org/dev/peps/pep-0008/">pep8</a> for example) unless absolutely required. What this will enable is, someone new joining the team would have less familiar things to get onboarded to making them productive faster in terms of moving around the codebase, finding things or making sense of why for certain choices.</p>
<p>Wrote a small thread around the same here.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Was talking to a friend of mine who was pining about that their PR's would sometimes get a lot of nits in terms of style guides. (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1338507609758859264?ref_src=twsrc%5Etfw">December 14, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="going-over-the-whole-design-again">Going over the whole design again</h2>
<p>A higher level design change is not something which I will suggest on a PR, this needs to be caught right before someone starts implementing foo feature/refactoring. It’s a massive waste of time for the whole team, while you go over something which neither of the team members had consensus upon. Which is why I really like the approach of <a href="https://en.wikipedia.org/wiki/Request_for_Comments">RFC</a> like process where the changeset if big enough would involve a discussion and a consensus be formed upon, so that things don’t come as surprises at the implementation stage for both the reviewer and the driver of the changeset.</p>
<p>What this will also immediately do is also make others familiar with what you are trying to propose and point out any mistakes which might not get caught by you in the early design phase, which they might have seen/experienced.</p>
<h2 id="idealistic-changeset">Idealistic changeset</h2>
<p>This is something which I would like to prevent in the codebase, if the feature adoption is hard for foo feature, it would probably not make sense to add it if no one is going to use it. There should be some strategy on how to get users for the problem you are solving and reducing the friction for adoption. If no-one is using that feature/it provides no added value in the codebase, it’s as good as dead code to me.</p>
<p>If the feature is not gonna get used, why bother adding it at that moment? If instead the attention can be diverted to more burning issues/solving customer problems which are gonna give more leverage.</p>
<p>Again, going via an RFC approach might be one way where this would get caught as others would be able to point out the shortcomings in planning on how this changeset is going to get adopted.</p>
<p>For example, there was this one change made in the codebase in the client of an API which would get distributed to developers internally for a toolset which our team would provide. This change would effectively introduce parsing of the API response and being quite tied to the exact semantics of the payload which it would recieve from the server. Trickling down such logic to the client meant, that any changes to the response would have to be backward compatible. And when we did end up introducing a change to the response payload, we ended up having to keep this versioned API backward compatible. While I agree that versioning is for this specific purpose, but I feel it could have been avoided if the client would have been decoupled from the exact response semantics in this case and would just about be very naive, instead the server being smart enough to send the appropriate data over to the client.</p>
<h2 id="accepting-a-large-changeset-which-introduces-too-many-changes">Accepting a large changeset which introduces too many changes</h2>
<p>With all honesty, I really don’t feel that someone would be able to effectively, actually go through a PR which changes 50 different files and has a huge LOC changeset, without having to spend copious amounts of time dedicated to just reviewing the PR. I personally have felt that such changes are super hard to review and would considerably increase the PR review time. In worst cases, to just unblock the team member, you have to either trust their changeset while you have gone through it on a very high level and going ahead and merging their PR, this introduces the problem of the PR not having gone through the usual PR process which would be a bit more rigorous.</p>
<p>There is no right size for a PR, but ideally smaller changes, which have one purpose are easier to review. Say for example, refactoring a flow to remove redundancy and re-using another module from the same codebase. Adding a small feature which would not require a lot of work.</p>
<p>If the feature/changeset requires a lot of changes, it would either mean that the scope was not broken down properly, which ended up creeping inside the PR as a reflection.</p>
<p>Over time people will notice what is a big enough changeset for their specific services and they will start breaking the PR’s down, if that is not happening it’s definitely something which needs to be brought to notice by other members in the team.</p>
<p>Another common thing, as pointed out by <a href="https://twitter.com/hashfyre/">Joy</a> is that we sometimes tend to add something extra in addition to the original scope of the PR. This becomes a problem when the scope of change, if too big/unrelated to the original scope, would affect the review of the PR as a sideeffect of the reviewer then trying to review two different contexts/intents. Keeping the intent to just one thing helps the reviewer’s job.</p>
<h2 id="being-pedantic-about-coverage">Being pedantic about coverage</h2>
<p><a href="https://en.wikipedia.org/wiki/Code_coverage">Code coverage</a> is not really a good metric to measure software and it’s stability. While good (lesser bugs) software tend to have high coverage, it’s not necessary that it’s gonna solve all the problems you would have/encounter. I have written a bit more about this <a href="https://www.tasdikrahman.com/2020/10/07/why-I-chose-to-do-tdd-for-my-side-project/">here</a> on why even close to 100% code coverage would not prevent your software to be bug free.</p>
<p>What I would like seeing is ways in which the changeset can be tested to gain more confidence of what is being introduced. The easier to test, the better as it would reduce the feedback cycle.</p>
<h2 id="ending-notes">Ending notes</h2>
<p>These are some of the prompts which I usually try following, which would allow the changeset getting accepted in a timely manner without compromising on the quality too much, but would love to hear what you think about it.</p>
<h2 id="credits">Credits</h2>
<p>Thanks to <a href="https://twitter.com/hashfyre/">Joy</a> and <a href="https://twitter.com/captn3m0/">nemo</a> for proof reading the post.</p>
To self host or to not self host kubernetes cluster(s)
2020-11-27T00:00:00+00:00
https://www.tasdikrahman.com/2020/11/27/to-self-host-or-to-not-self-host-your-kubernetes-cluster
<p>A friend of mine asked this to me recently, about how was it to <a href="https://en.wikipedia.org/wiki/Self-hosting">self host</a> <a href="https://kubernetes.io">kubernetes</a> clusters. And I was cursing myself about why I did not write this post earlier (I mean, technically I have written about how we used to do self hosting <a href="https://www.tasdikrahman.com/2019/04/04/self-hosting-highly-available-kubernetes-cluster-aws-google-cloud-azure/">before</a>, but not the pros and cons of it), as this was not the first time I had been asked this question. So this post is dedicated to my friend and to others when they chance upon this question.</p>
<p>Just for context, self host here does not only mean, <a href="https://www.tasdikrahman.com/2019/04/04/self-hosting-highly-available-kubernetes-cluster-aws-google-cloud-azure/">kubernetes inside kubernetes</a>, but in a broader sense, would mean, managing the control plane of the kubernetes cluster too, along with the worker nodes (which is the usual case these days with the cloud vendor k8s options). You would preferably be using solutions like <a href="https://github.com/kubernetes/kops">kops</a>, or <a href="https://github.com/poseidon/typhoon">typhoon</a> here to self host your kubernetes clusters. But there are a bunch of really great tools out there these days apart from these two, which I personally like.</p>
<p>I had to pry out this tweet thread which I wrote about sometime back and it’s a good summary more or less.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">If you can, don't try managing your own <a href="https://twitter.com/kubernetesio?ref_src=twsrc%5Etfw">@kubernetesio</a> clusters. It can become a huge engineering effort in itself very quickly. If the core business product is not around providing infrastructure to others, using <a href="https://twitter.com/hashtag/kubernetes?src=hash&ref_src=twsrc%5Etfw">#kubernetes</a> solutions provided by cloud providers is not a bad idea</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1110927284926447617?ref_src=twsrc%5Etfw">March 27, 2019</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h3 id="dont-self-host-unless-you-have-a-very-good-reason-to">Don’t self host unless you have a very good reason to</h3>
<p>The very obvious answer if you ask why, is because with all honesty, it’s a huge engineering effort to even run and manage vendor managed kubernetes clusters, let alone self hosted ones. I have already written quite a lot, on a bunch of reasons, one particular problem, from which there’s no running away away from, is kubernetes upgrades, which I have covered in detail <a href="https://www.tasdikrahman.com/2020/07/22/a-few-notes-on-gke-kubernetes-upgrades/">here</a> on what entails in one such upgrade, for you to understand the complexity of things.</p>
<p>And there are a bunch of things, which I have mentioned in this <a href="https://www.tasdikrahman.com/2020/11/21/choosing-between-one-big-cluster-or-multiple-smaller-kubernetes-clusters/">post</a> on the complexities which entail with managing a kubernetes cluster, vendor managed or self-hosted alike and how effort multiplies as one goes into the direction of multiple clusters and different cluster sizes. If you have read the above two posts, you might just feel this post getting a bit repeated at this point, hence I will not delve into the exact details, as I have covered them in depth in the other two posts.</p>
<h3 id="but-tasdik-i-need-to-customize-my-kubernetes-installation-i-will-lose-that-power-if-i-go-ahead-with-a-vendor">But Tasdik, I need to customize my kubernetes installation, I will lose that power if I go ahead with a vendor</h3>
<p>A lot of flexibility in terms of customisations, in a vendor managed kubernetes cluster is alreay being provided these days. But even then, if you are really trying to use/set something esoteric which is really important to you and is required for your workloads, do keep in mind the maintenance of the cluster. If you do end up self-hosting it, you would need to know how to operate it and the propblems which come and the corner/edge cases with it.</p>
<p>An event like switching out our <a href="https://chrislovecnm.com/kubernetes/cni/choosing-a-cni-provider/">CNI</a>, moving from kube-dns to <a href="https://kubernetes.io/docs/tasks/administer-cluster/coredns/">core-dns</a>, for example, is best abstracted if possible.</p>
<p>I get that people do run other things like postgresql or mysql on VM’s, but postgresql doesn’t need an upgrade every few months, and would just about run for years without an upgrade without too many hiccups for most part.</p>
<p>And not to forget kubernetes is a really fast moving project. <a href="https://github.com/kubernetes/kubernetes/releases/tag/v1.4.6">v1.4.6</a> was released around in november, 2016. It was also a version (~1.8 to be precise) close to which we started running our production workloads back in 2017 for an org. Come 2020 november, we already have <a href="https://github.com/kubernetes/kubernetes/releases/tag/v1.18.12">v1.18.12</a>, and LTS not a thing as of now in the kubernetes world. And not to mention the amount of changes which happen each release. As with any fast moving project, things are bound to change, as <a href="https://www.infoq.com/podcasts/joe-beda-kubernetes-cncf/">Joe</a> mentions here that kubernetes should be boring. Kubernetes will take some time, to become something similar to that. On a related note, this is a very good piece on that <a href="https://mcfunley.com/choose-boring-technology">topic</a>.</p>
<p>There’s just a mouthful of things to take care of too, in case you decide to manage your own <a href="https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/">etcd clusters</a>, along with the control plane components, if you decide to self host. There’s great learning for sure in all this, but the idea is to provide a reliable peice of infrastructure over to your customers here first (assuming you work in the operations/platform team, your customers are your developers), unless the team is really rock solid in terms of their k8s knowledge and is able to keep up with self hosting, this sort of path can very quickly become a possible wrong decision for your team.</p>
<p>And not to forget the automation which needs to be maintained to bring up/upgrade parts if not all of the whole self-hosted cluster automation. The automation can vary based on the solution which you picked up to orchestrate the creation of your cluster and could be anything from shell scripts/terraform modules etc. The point is that, if it’s not a standard solution out there and you have hand smashed a lot of automation yourself, the added overhead of maintaing that automation shows up over time.</p>
<h3 id="ending-notes">Ending notes</h3>
<p>Unless providing kubernetes as a PaaS is your bread and butter and your main business, where you compete against others and have to absolutely innovate and stay on top of the game. In most cases you are better off just using a cloud vendor managed offering.</p>
<p>We ended up self-hosting back in the days, as the cloud vendored k8s offering was not present in our region and due to constraints of compute, being present on that particular region and no particular one, made us go the self-hosting route.</p>
<p>Having managed(both self hosted and vendor managed) and built platforms on top of multiple k8s clusters(cloud vendor managed control plane in this case), for the better part of 3 and a half years now, one thing for sure I love about k8s is the power it gives. The ease of deployments(<a href="https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/">rolling deployments</a> out of the box), the immutable nature of deployments(containerized workloads), the programmatic interface for starters(<a href="https://github.com/kubernetes/client-go">kubernetes/client-go</a>, need I even say more) and I am not even scratching the surface.</p>
<p>But at the same time, the right tradeoffs do need to made. Kubernetes absolutely solves some very core problems, but if you’re just in the phase of building your product, with not much of bandwidth. Would highly recommend thinking twice in case of going the self hosting route or even kubernetes, if you must for it to not become a distraction from the core task, which is reliable infrastructure. You might just be better off with plain simple <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html">ASG’s</a> and <a href="https://cloud.google.com/compute/docs/instance-groups">instance groups</a> for that matter with <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">AMI’s</a>. Less moving part is good to start with. But again, depends on your team, if everyone has a good amount of production experience with k8s, then why not? But even then, vendor managed k8s installation will again be my personal pick.</p>
<p>If you are in a very specialised environment where you absolutely know what you are doing and you must, this post is obviously not for you, in which case I would love to hear more about your learnings :).</p>
<p>Coincidentally, my last post was also about the trade-offs in specific situation similar to this situation, when dealing with kubernetes. Hope you have found this piece informative.</p>
<h3 id="links">Links</h3>
<ul>
<li><a href="https://www.tasdikrahman.com/2020/07/22/a-few-notes-on-gke-kubernetes-upgrades/">https://www.tasdikrahman.com/2020/07/22/a-few-notes-on-gke-kubernetes-upgrades/</a></li>
<li><a href="https://www.tasdikrahman.com/2019/04/04/self-hosting-highly-available-kubernetes-cluster-aws-google-cloud-azure/">https://www.tasdikrahman.com/2019/04/04/self-hosting-highly-available-kubernetes-cluster-aws-google-cloud-azure/</a></li>
<li><a href="https://www.tasdikrahman.com/2020/11/21/choosing-between-one-big-cluster-or-multiple-smaller-kubernetes-clusters/">https://www.tasdikrahman.com/2020/11/21/choosing-between-one-big-cluster-or-multiple-smaller-kubernetes-clusters/</a></li>
</ul>
<h3 id="credits">Credits</h3>
<ul>
<li>Thanks to <a href="https://captnemo.in/">nemo</a> for proofreading and the folks at <a href="https://kubernetes.io/">kubernetes</a> for the post image.</li>
</ul>
Choosing between one big cluster or multiple smaller kubernetes clusters
2020-11-21T00:00:00+00:00
https://www.tasdikrahman.com/2020/11/21/choosing-between-one-big-cluster-or-multiple-smaller-kubernetes-clusters
<p>This post is a continuation on the discussion which I was having with <a href="https://twitter.com/VineethReddy02">@vineeth</a></p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">But why would someone choose one large cluster over multiple small clusters? Aren't multiple clusters already a pattern in enterprises?</p>— Vineeth Pothulapati (@VineethReddy02) <a href="https://twitter.com/VineethReddy02/status/1329656769900003329?ref_src=twsrc%5Etfw">November 20, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Context is when I came across a tweet which demonstrated the ability of kubernetes to scale uptill 15k nodes due to recent improvements.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">15k nodes cluster 🤯 <a href="https://t.co/VMWI7HeYHH">https://t.co/VMWI7HeYHH</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1329615066405093379?ref_src=twsrc%5Etfw">November 20, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>The discussion was originally around costs and how much would it take to run one such large kubernetes cluster, but it went into a different direction altogether.</p>
<p>So what should the decision be? Rather, I would like to take the path, where we discuss a few things about both sides of the coin. While this post is not a recommendation on what one should do, but the idea is to guide you to take a more informed decision with the data points, trade-offs and constraints that you have.</p>
<p>Before we start, big, here will be relative, but for the sake of this conversation, let’s say you plan to run a cluster, with worker nodes greater than ~50 (this is a big cluster for me right now, more on the why part of it in a bit) and a small cluster for the context of this post is say, 5-10 worker nodes.</p>
<h4 id="basic-house-keeping">Basic house keeping</h4>
<p>Having separate cluster(s) for staging and production, similar to how you would have been following the same for your compute infrastructure on VM’s.</p>
<h4 id="multi-tenancy">Multi tenancy</h4>
<p>Hard multi-tenancy is something, which might not work the way as expected as of now on k8s, even with <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">netpols</a>/<a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/">namespaces</a> and upcoming mechanisms like <a href="https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc">Hierararchical namespaces (aka HNC)</a>, which is still in incubation at the time of writing this.</p>
<p>That’s what I have last checked, but please correct me here if you came across something which tells otherwise.</p>
<p>So if your workloads do have a hard requirement for this, multiple k8s clusters would be a way here,</p>
<h4 id="upgrade-charter">Upgrade charter</h4>
<p>One operational aspect which is important to note here is the burden of upgrades. Keeping up with upgrades, with the release cycle of kubernetes is not a trivial task. I have written about our experiences in this thread sometime back, to give you an idea of what really goes into one such upgrade.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">A few notes on <a href="https://twitter.com/kubernetes?ref_src=twsrc%5Etfw">@kubernetes</a> cluster upgrades on GKE (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1285619368353726465?ref_src=twsrc%5Etfw">July 21, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Even on a managed platform provided by the cloud vendors (where they maintain the control plane for you), it’s still an operational heavy task, and we are not even talking about upgrades on self hosted and managed clusters via tools for eg: <a href="https://github.com/kubernetes/kops/">kops</a>.</p>
<p>Multiply this effort with multiple such clusters, and you will end up needing to have people dedicated to just do this. There is no such thing as an <a href="https://en.wikipedia.org/wiki/Long-term_support">LTS</a> release as of now, unless you are fine with running super old kubernetes installations which would have CVE’s reported and fixed in the upcoming releases/you are ok with doing big bang upgrades. Both of which might not be a great idea to begin with. Even if someone decides to run an archaic installation for long, if you feel running a super old installation will fly with your cloud provider, you will be in for a rude shock, where they can literally force upgrade your cluster(yes, they do it).</p>
<p>All of the toil which goes into cluster upgrades might have been a factor, which would have led to the initiation of this discussion <a href="https://github.com/kubernetes/sig-release/issues/1290">here</a> where folks discuss about modifying the kubernetes release cadence. I for one <a href="https://github.com/kubernetes/sig-release/issues/1290#issuecomment-709774428">did vote</a>, on moving to 3 releases than 4 every year. But there are also arguments against doing slower releases, which might make the next release bloated with features/deprecations and going against the philosophy of small incremental changes, but that discussion is for another post.</p>
<p>Given someone ends up with a large cluster, the upgrade is equally gonna be as operationally heavy, if not more than when someone was upgrading a bunch of clusters, as the operations to be done would remain more or less the same.</p>
<p>What might increase a side effect of the cluster being huge, is that someone would have to baby sit the whole upgrade operation longer, than what they would spend on a smaller cluster(given upgrade automation is not present/not mature enough to remove the human involved here). The upgrades here, have to be done one ASG group/node pool, at a time, and takes time even if done for a smaller cluster. Imagine having to do it with a large cluster size, having 100’s of nodes in each node pool. But I am sure there are ways in which someone would have improvised here, but I am yet to come across something like that.</p>
<p>The eventual state is to have automation which does a bunch of domain specific operations, and the human has to only get paged/notified in case when the upgrade gets done and dusted/there was some failure which occured, which the decision tree wasn’t able to resolve itself. <a href="https://github.com/keikoproj/">Keiko</a> project’s <a href="https://github.com/keikoproj/upgrade-manager">upgrade manager</a> is one such tool which comes to mind, apart from <a href="https://github.com/hellofresh/eks-rolling-update">https://github.com/hellofresh/eks-rolling-update</a>.</p>
<p>And if there are non-standardized workloads in the cluster(folks hand applying yamls/no track of objects in cluster), there are various ways on how an upgrade can either get stuck due to pdb budgets of pods not being met when the node gets cycled/end up causing an outage. Which makes this whole operation for a focused cluster running x teams applications much more saner.</p>
<p>Partly due to the fact that, now if the cluster is owned by x team, you can at least ask them to be on call with you during the upgrade process, so that they can at least shadow the person doing the upgrade and monitor the business metrics of the team’s products being served out of the cluster being upgraded.</p>
<p>Hence, upgrade strategy is something to consider seriously while choosing the multiple clusters vs single cluster strategy.</p>
<h4 id="api-deprecations">API deprecations</h4>
<p>API’s getting deprecated, for example <a href="https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/">1.16</a> was a release which had a bunch of changes, which would have involved the cluster operators to upgrade their automation/helm charts to be compatible with those changes, if they had plans on upgrading from some x version to v1.16.</p>
<p>There’s no one to blame here in case of API deprecations, an object getting stabler and getting promoted to a more stable API, is a natural progression. To enjoy the benefits of a stable kubernetes object, it only makes sense to move to the stable api rather than being stuck on one which is less stable/getting deprecated in the next release.</p>
<p>For a larger org having bandwidth, managing multiple clusters for teams, they will eventually have automation over time to reduce the toil.</p>
<p>But if you’re a small org, with a small group of folks managing the k8s clusters, the manual toil will be quite high. The reasons are also obvious, the lack of bandwidht will attribute to them not being able to automate the redundant tasks required for the upgrade. Even if they manage to write some automation, the automation will become stale over time if not given prioritisation to maintain it, as the domain changes. Plus, an average operations team will also have developer requests coming in their way, prioritising all this along with tasks such as maintaing your k8s cluster? Definitely a hard task to begin with.</p>
<p>Although I have not personally used it, but I have heard good things about <a href="https://github.com/doitintl/kube-no-trouble">kube-no-trouble</a>, which takes a stab at telling the deprecated API’s in a cluster.</p>
<h4 id="access-management">Access management</h4>
<p>Giving the right kind of access to the developers/operational folks/xyz person in the team is necessary problem to solve, unless you are giving admin privileges to only the operational folks/one groups of people. Even then, solving the same problem over for multiple clusters requires automation and a proper mechanism.</p>
<p>What is to be done when a person leaves/joins? How can access be granted in granular manner for certain rbac roles which are only to be given read access, but no deletion access. If you have been in a position, where someone from the team has deleted the <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">deployment object</a> of your service and you ended up with an outage. This can happen with anyone, but reducing the blast radius is not a bad idea.</p>
<p>If your workloads are sensitive (payments data etc.), you would require access for such environments to be tighly audited and managed.</p>
<p>There are a few tools out there which would help you in doing so, like <a href="https://github.com/appvia/krane">krane</a>, <a href="https://github.com/appvia/krane">audit2rbac</a>, which help you in this process (thanks to <a href="https://rmenn.in">rahul</a> for pointing them out to me).</p>
<h4 id="deployment-automation">Deployment automation</h4>
<p>With all honesty, hand applying yaml files will only go so far. It works, but is a recipe for disaster over the long run, creating more spaghetti in the cluster and creating more problems than solving them. Problems like, who applied x resource object, who changed x resource, who is using this x resource. And the list so goes on. Multiple clusters or a single large cluster doesn’t matter here in this context, but someone editing something which they are not supposed to for some other teams product?</p>
<p>Namespace as a service, to teams, is a model which I have heard people doing, keeping all access to specific roles tied to a particular namespace.</p>
<p>Continous Delivery is hardly a requirement for anyone(mostly?), but the ability to reliably deploy something is a hard requirement in most cases though.</p>
<p>Having some form of CI, to safely modify the respective resources via a tool/process will prevent a lot of surprises in production. Audit trail logs in the CI pipeline for someone to see what happened, automated rollbacks too? That would be a sweet spot I would say. (We have built out our deployment platform similar to these practices described in my current org, but more on that in another post)</p>
<p>There are multiple tools out there which allow fine grained RBAC rules, CI/CD, progressive delivery to your kubernetes clusters, <a href="https://flagger.app/">flagger</a>, <a href="https://argoproj.github.io/argo-cd/">argo</a> being a few ones to name.</p>
<h4 id="concentration-risk">Concentration risk</h4>
<p>If we look at the flip-side, having one big cluster, concentrates the risk of failure.</p>
<p>If you are not on a vendor specific kubernetes installation, what happens when the control plane goes for a toss? 1 person having all the context is not scalable, what happens when the person who knows the operational know-how, to fix x problem, is not present to handle the pager?</p>
<p>What about zonal failures affecting the cluster? Do we have checks and balances to handle such an event?</p>
<p>What if the <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits">requests and limits</a> for memory and CPU were not set properly for foo service in foo namespace, that it ended up hogging more cpu and memory, affecting baz service from baz team(baz service’s getting degraded). It’s not a great place to be in. Overprovisioning pods for teams/services and then gradually checking the trendline usage over time and then setting them over in requests and limits is one way to have sanity in the cluster and sizing it accordingly. More on <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/">resource quotas</a> in another post.</p>
<p>Having one cluster, per product group is also a model which people follow, helping de-risk the affects of failure to not affect other products/product groups. But then the operation problem/complexity of managing multiple clusters arise.</p>
<h4 id="ending-notes">Ending notes</h4>
<p>Either of the two options, one large cluster vs multiple smaller clusters, both are an opinionated way to run clusters, or for that matter any compute infrastructure. What works best, might not work out in another context. Someone else’s best practice might turn out into a nightmare for another org/team to manage/run.</p>
<p>Given, most/all of these problems can be solved, the right trade-offs can be made when deciding for a solution and I hope this discussion helps you making the right decision in your context.</p>
<p>P.S. A good amount of discussion also happened on <a href="https://reddit.com/r/kubernetes">/r/kubernetes</a>, here in <a href="https://www.reddit.com/r/kubernetes/comments/jy85ly/choosing_between_one_big_cluster_or_multiple/">this thread</a></p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">https://kubernetes.io/docs/concepts/services-networking/network-policies/</a></li>
<li><a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/">https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/</a></li>
<li><a href="https://blog.gojekengineering.com/how-we-upgrade-kubernetes-on-gke-91812978a055">https://blog.gojekengineering.com/how-we-upgrade-kubernetes-on-gke-91812978a055</a></li>
<li><a href="https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/">https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/</a></li>
<li><a href="https://github.com/kubernetes/sig-release/issues/1290">https://github.com/kubernetes/sig-release/issues/1290</a></li>
<li><a href="https://github.com/kubernetes-sigs/multi-tenancy">https://github.com/kubernetes-sigs/multi-tenancy</a></li>
<li><a href="https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc">https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc</a></li>
<li><a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits">https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits</a></li>
<li><a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md">https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md</a></li>
</ul>
Testing rake tasks with rspec
2020-10-20T00:00:00+00:00
https://www.tasdikrahman.com/2020/10/20/testing-rake-tasks-with-rspec
<p>This blog post is a continuation of this thread.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">On trying to write a spec for one of the rake tasks, when trying to invoke the same rake tasks within the same <a href="https://twitter.com/rspec?ref_src=twsrc%5Etfw">@rspec</a> contexts, for different flows, weirdly the tests failed if I ran the whole suite, but would pass if I ran them separately.</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1293581788455952384?ref_src=twsrc%5Etfw">August 12, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>So for example</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># ./lib/tasks/foo_task.rake</span>
<span class="n">desc</span> <span class="s1">'Foo task'</span>
<span class="n">namespace</span> <span class="ss">:task</span> <span class="k">do</span>
<span class="n">task</span> <span class="ss">:my_task</span><span class="p">,</span> <span class="p">[</span><span class="ss">:foo</span><span class="p">,</span> <span class="ss">:bar</span><span class="p">]</span> <span class="o">=></span> <span class="p">[</span><span class="ss">:baz</span><span class="p">]</span> <span class="k">do</span> <span class="o">|</span><span class="n">task</span><span class="p">,</span> <span class="n">args</span><span class="o">|</span>
<span class="o">...</span>
<span class="c1"># does my_task</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now if we try writing a spec a for it</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># ./spec/tasks/foo_task_spec.rb</span>
<span class="nb">require</span> <span class="s1">'rails_helper'</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">load_tasks</span>
<span class="n">describe</span> <span class="s2">"task_my_task"</span> <span class="k">do</span>
<span class="n">context</span> <span class="s2">"foo case"</span> <span class="k">do</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg1</span><span class="p">)</span> <span class="p">{</span><span class="s2">"foo"</span><span class="p">}</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg2</span><span class="p">)</span> <span class="p">{</span><span class="s2">"baz"</span><span class="p">}</span>
<span class="n">it</span> <span class="s2">"it does foo behaviour"</span> <span class="k">do</span>
<span class="no">Rake</span><span class="o">::</span><span class="no">Task</span><span class="p">[</span><span class="s2">"task:my_task"</span><span class="p">].</span><span class="nf">invoke</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">)</span>
<span class="c1"># assert the expected behaviour here related for foo case</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">context</span> <span class="s2">"baz case"</span> <span class="k">do</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg1</span><span class="p">)</span> <span class="p">{</span><span class="s2">"bazbee"</span><span class="p">}</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg2</span><span class="p">)</span> <span class="p">{</span><span class="s2">"foobee"</span><span class="p">}</span>
<span class="n">it</span> <span class="s2">"it does baz behaviour"</span> <span class="k">do</span>
<span class="no">Rake</span><span class="o">::</span><span class="no">Task</span><span class="p">[</span><span class="s2">"task:my_task"</span><span class="p">].</span><span class="nf">invoke</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">)</span>
<span class="c1"># assert the expected behaviour here related for baz case</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now if we try to run the specs for specific contexts, where the rake task is being invoked, all will work well, but when we try to run the specs for all the contexts in the test file, the first rake task will run, but the rest of them will start failing.</p>
<p>Which is confusing, turns out that the tasks can be invoked only once in a given context. Not sure of the history behind this or the reasoning on why this is the case, couldn’t find it. (let me know if you were able to get this bit, would be happy to learn it’s history)</p>
<h3 id="how-to-make-it-work">How to make it work</h3>
<p>To work around this, the task needs to be explicitly re-enabled in before the next spec is run.</p>
<p>Addin an <code class="language-plaintext highlighter-rouge">after_each</code> block would work for starters, if inside that block you would re-enable your task which you are trying to test out in your spec, which would mean that after each spec, inside which you are exercising your method, this routine is called. So something like</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># ./spec/tasks/foo_task_spec.rb</span>
<span class="nb">require</span> <span class="s1">'rails_helper'</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">load_tasks</span>
<span class="n">describe</span> <span class="s2">"task_my_task"</span> <span class="k">do</span>
<span class="n">after</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Rake</span><span class="o">::</span><span class="no">Task</span><span class="p">[</span><span class="s2">"task:my_task"</span><span class="p">].</span><span class="nf">reenable</span>
<span class="k">end</span>
<span class="n">context</span> <span class="s2">"foo case"</span> <span class="k">do</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg1</span><span class="p">)</span> <span class="p">{</span><span class="s2">"foo"</span><span class="p">}</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg2</span><span class="p">)</span> <span class="p">{</span><span class="s2">"baz"</span><span class="p">}</span>
<span class="n">it</span> <span class="s2">"it does foo behaviour"</span> <span class="k">do</span>
<span class="no">Rake</span><span class="o">::</span><span class="no">Task</span><span class="p">[</span><span class="s2">"task:my_task"</span><span class="p">].</span><span class="nf">invoke</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">)</span>
<span class="c1"># assert the expected behaviour here related for foo case</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">context</span> <span class="s2">"baz case"</span> <span class="k">do</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg1</span><span class="p">)</span> <span class="p">{</span><span class="s2">"bazbee"</span><span class="p">}</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:arg2</span><span class="p">)</span> <span class="p">{</span><span class="s2">"foobee"</span><span class="p">}</span>
<span class="n">it</span> <span class="s2">"it does baz behaviour"</span> <span class="k">do</span>
<span class="no">Rake</span><span class="o">::</span><span class="no">Task</span><span class="p">[</span><span class="s2">"task:my_task"</span><span class="p">].</span><span class="nf">invoke</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">)</span>
<span class="c1"># assert the expected behaviour here related for baz case</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">reenable</code> first resets the task’s <code class="language-plaintext highlighter-rouge">already_invoked</code> state, allowing the task to then be executed again with all it’s dependencies.</p>
<p>There are <code class="language-plaintext highlighter-rouge">execute</code> and <code class="language-plaintext highlighter-rouge">invoke</code> methods too which are mentioned in this <a href="https://stackoverflow.com/a/32382929">SO answer</a>. No preference as such to why I went with <code class="language-plaintext highlighter-rouge">reenable</code>.</p>
<p>Running all the specs would work now.</p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://relishapp.com/rspec/rspec-core/v/2-2/docs/hooks/before-and-after-hooks">https://relishapp.com/rspec/rspec-core/v/2-2/docs/hooks/before-and-after-hooks</a></li>
<li><a href="https://ruby-doc.org/stdlib-2.0.0/libdoc/rake/rdoc/Rake/Task.html#method-i-reenable">https://ruby-doc.org/stdlib-2.0.0/libdoc/rake/rdoc/Rake/Task.html#method-i-reenable</a></li>
<li><a href="https://stackoverflow.com/questions/32381017/running-rake-tasks-in-rspec-multiple-times-returns-nil">https://stackoverflow.com/questions/32381017/running-rake-tasks-in-rspec-multiple-times-returns-nil</a></li>
</ul>
<h3 id="credits">Credits</h3>
<ul>
<li>Picture credits to <a href="https://rspec.info/">https://rspec.info/</a></li>
</ul>
A few things about database migrations
2020-10-18T00:00:00+00:00
https://www.tasdikrahman.com/2020/10/18/a-few-things-about-database-migrations
<p>This blog post is a continuation of these two threads.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">A few things about database schema changes. (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1317490663613624320?ref_src=twsrc%5Etfw">October 17, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">This is where <a href="https://twitter.com/rails?ref_src=twsrc%5Etfw">@rails</a> active record migrations really shine. I find it's UX super clean. (1/n)<a href="https://t.co/vA6Jb345yc">https://t.co/vA6Jb345yc</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1317695102924451840?ref_src=twsrc%5Etfw">October 18, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>The schema of your relational database, will change over time for your application. Trying to introduce these changes from dev setup -> integration/UAT -> production env, in a clear, consistent and repeatable manner, would definitely add value in trying to maintain repeatability across different environments.</p>
<h2 id="ways-to-introduce-changes-to-your-database">Ways to introduce changes to your database</h2>
<p>If someone is introducing the schema changes manually in these environments, tracking such changes and reproducing them becomes a task.</p>
<p>What about auditing what ran, or what was changed? Who introduced it? What happens when a schema change was made and you were not aware of it in the integration env (fill anything else here for that matter), which is also making your current build fail, but you have already wasted some time trying to debug it?</p>
<p>The time lost in debugging such issues, could be utilized somewhere else. The other thing to note here being, that using a single database for the integration environment, where multiple developers will deploy their codebases and introduce changes to the schema, is one way or the other, gonna bite the team down the line.</p>
<p>The ability to be able to recreate the structure of your database, consistently across local and dev environments will allow to iterate faster for sure. Having all the DDL and DML scripts, with a sequence, checked into your VCS allows one to at least track what has been run.</p>
<h2 id="tracking-how-your-schema-is-evolving">Tracking how your schema is evolving</h2>
<p>But now how do you track which DDL/DML scripts ran for a particular environment and it’s database?</p>
<p>A very simple implementation to solve this is described <a href="http://blog.cherouvim.com/a-table-that-should-exist-in-all-projects-with-a-database/">here</a> where you have a table specifically to track this,which records the sequence of the DDL/DML script, which last ran, the time when it was run & a small description mentioning what is it doing.</p>
<p>And whenever someone is running any DDL/DML command, they would also need to insert into this table, helping in keeping track what’s the last status of the database for that environment.</p>
<p>There are much more mature & well tested ways to do this, like active record migrations in <a href="https://rubyonrails.org">rails</a>, <a href="https://flywaydb.org/">Flyway</a> in java world, <a href="https://github.com/golang-migrate/migrate/">golang-migrate</a> in golang et al. The root of it being, having a way to know what ran where & having a repeatable way to setup the schema of your database across different environments.</p>
<p>This is where rails active record migrations really shine. I find it’s UX super clean.</p>
<p>All migration files, along with the schema of your database is also checked into the VCS along with your business logic, prefixed with a UTC timestamp, the timestamp being in the <code class="language-plaintext highlighter-rouge">schema_migrations</code> table, which active record would maintain inside your database.</p>
<p>The idea is the same. Track the status of which DDL/DML scripts have run for your application.</p>
<p>Want to see, what’s the status of your database and what has run on it? <code class="language-plaintext highlighter-rouge">rails db:migrate:status</code> is gonna give you all the status of all the migration scripts and the status of whether it has already been applied or not.</p>
<p>Want to roll back the migration which was run on your database? <code class="language-plaintext highlighter-rouge">rails db:rollback STEP=n</code>, to rollback n versions of the migration id which have already been applied. The n being any integer value, of the number of migration files you want to roll back by.</p>
<p>Want to redo a migration which was rolled back/run it again? <code class="language-plaintext highlighter-rouge">rails db:migrate:redo VERSION=<UTC-timestamp-prefix-of-migration-file></code>.</p>
<p>A similar approach can be found in <a href="https://github.com/golang-migrate/migrate">golang-migrate</a>, where each schema change is introduced with an SQL script numbered sequentially starting from 1 for example (the 1 can be any 64 bit unsigned integer), and ending with a suffix of up.sql and down.sql</p>
<p>Each new schema change will be added in a new SQL file, and numbered accordingly. Users would then add the helper methods provided by golang-migrate, for running migrations into the cli interface for their app, for both applying the migrations and to rollback them.</p>
<p>Both the examples, we can infer that the central theme is the same, which is to keep track of what ran and what did has not run along with a way, either via timestamps or via simple count to keep track of the order of the migration scripts</p>
<p>Another thing to notice here is that, both of them encourage you/give you mechanisms to write the schema change such that it is possible to reverse it to a previous state.</p>
<h2 id="constraints-which-can-be-put-on-the-database">Constraints which can be put on the database</h2>
<p>Rails allows one to introduce validations on models before persisting the object to the database. But it’s also important to have the same validation wherever you can on your schema. Allowing both the ORM and the database to enforce validations.</p>
<p>The model.<a href="https://apidock.com/rails/ActiveRecord/Persistence/update">update</a> method, does this for you. It will do the validations and the callbacks required for example, along with updated_at/updated_on for you, whenever you are trying to update the attributes for the object.</p>
<p>This will immediately help, in having the 1st-order check to prevent inserting something you shouldn’t have. The 2nd-order check being in your database schema itself. Being present as your final guard for the entries to not be dirty. While I don’t remember anyone telling that it is a rule of thumb to have both, but it’s not either uncommon to stumble upon this either. You may argue that it goes against <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>, but I feel having the final validation on the schema of the database, definitely acts as the final source of truth where you can always fallback on and there’s no downside to it.</p>
<p>A very simple example of this can be, when you are trying to put a check on a column in your database to ever not be a null value, even if your model has a validation to protect against inserting a null value, it can be backed by a database constraint at the same time for the database for enforce this at the end too along with the ORM.</p>
<p>There are also methods like <a href="https://apidock.com/rails/ActiveRecord/Persistence/update_column">update_column</a>, which will straight up update the attribute which you want to insert to the database, skipping all the validations, callbacks etc. Don’t use it in your DML script, as part of a migration file, unless you have a good reason to.</p>
<h2 id="rolling-out-migrations-for-your-applications">Rolling out migrations for your applications</h2>
<p>As to how to roll out migration changes to the end application? I feel since the state changes over schema changes, it is similar to doing a new deployment for your application</p>
<p>For example, if you are dropping a column from a table, the code changes should go first, where you remove reference to that table everywhere and then do a deploy. Do your e2e tests, check if everything is working and then in the next deployment, dropping that column.</p>
<p>As with all changes, I feel iterative and small changes are always easier to make sense of or debug if an issue arises, given the changelog will be easier to grok through and pin-point easily what could have introduced the issue.</p>
<p>As I learn more on this, I would love to hear on resources to read more on this topic/processes/techniques which you folks follow, which have worked well for you.</p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://web.archive.org/web/20111115150123/http://blog.stelligent.com:80/integrate-button/2010/07/database-integration-in-your-build-scripts.html">https://web.archive.org/web/20111115150123/http://blog.stelligent.com:80/integrate-button/2010/07/database-integration-in-your-build-scripts.html</a></li>
<li><a href="https://odetocode.com/blogs/scott/archive/2008/01/30/three-rules-for-database-work.aspx">https://odetocode.com/blogs/scott/archive/2008/01/30/three-rules-for-database-work.aspx</a></li>
<li><a href="https://odetocode.com/blogs/scott/archive/2008/01/31/versioning-databases-the-baseline.aspx">https://odetocode.com/blogs/scott/archive/2008/01/31/versioning-databases-the-baseline.aspx</a></li>
<li><a href="http://blog.cherouvim.com/a-table-that-should-exist-in-all-projects-with-a-database/">http://blog.cherouvim.com/a-table-that-should-exist-in-all-projects-with-a-database/</a></li>
<li><a href="https://dba.stackexchange.com/questions/2/how-can-a-group-track-database-schema-changes">https://dba.stackexchange.com/questions/2/how-can-a-group-track-database-schema-changes</a></li>
<li><a href="https://github.com/golang-migrate/migrate/blob/master/MIGRATIONS.md">https://github.com/golang-migrate/migrate/blob/master/MIGRATIONS.md</a></li>
<li><a href="https://edgeguides.rubyonrails.org/active_record_migrations.html">https://edgeguides.rubyonrails.org/active_record_migrations.html</a></li>
<li><a href="https://www.martinfowler.com/articles/evodb.html">https://www.martinfowler.com/articles/evodb.html</a></li>
<li><a href="https://thoughtbot.com/blog/validation-database-constraint-or-both">https://thoughtbot.com/blog/validation-database-constraint-or-both</a></li>
</ul>
<h3 id="credits">Credits</h3>
<ul>
<li>Cover image credits to Shripal Dapthary. <a href="https://unsplash.com/photos/3BbEYiIV7bo">Source</a></li>
</ul>
The making of bhola - your cert expiration overseer - Part 1
2020-10-08T00:00:00+00:00
https://www.tasdikrahman.com/2020/10/08/the-making-of-bhola-your-cert-expiration-overseer-part-1
<p>You might have already seen me writing a bit about bhola already on <a href="https://twitter.com/tasdikrahman">twitter</a>, I wrote a little bit about why I have been building <a href="https://github.com/tasdikrahman/bhola">bhola</a>. This post is more of a continuation to this tweet and what I envision it to be moving forward.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Do you sometimes wake up, with a call by someone from your team, telling you some SSL cert has expired? Do you keep track of SSL cert expirations on your to do notes or excel sheets? Would you like to be on top of such x509 cert renewals? <a href="https://t.co/MVFRZCUlZN">https://t.co/MVFRZCUlZN</a> is for you (1/n) <a href="https://t.co/pj8JHJEkje">pic.twitter.com/pj8JHJEkje</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1306945863369936896?ref_src=twsrc%5Etfw">September 18, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="what-was-the-inspiration">What was the inspiration</h2>
<p>Do you sometimes wake up, with a call by someone from your team, telling you some SSL cert has expired? Do you keep track of SSL cert expirations on your to do notes or excel sheets? Would you like to be on top of such x509 cert renewals?</p>
<p>All of this are directly due to</p>
<ul>
<li>No visibility about when the certificate is expiring</li>
<li>No alerts in form of email or test message when the certificate is expiring</li>
</ul>
<p>Then <a href="https://github.com/tasdikrahman/bhola">bhola</a> is for you!</p>
<h2 id="what-does-bhola-do">What does bhola do?</h2>
<p><a href="https://github.com/tasdikrahman/bhola/releases/tag/v0.1.0">v0.1</a> of Bhola, gives you a dead simple API, which you can use to ask Bhola, to track domains which have certs attached to it. It automatically checks for the cert expiration in the background keeping note of when is it expiring.</p>
<p>The operator can set a buffer period, which would bhola, then use to see if it meets the threshold number of days, before the cert is going to expire, before marking the cert, that it needs renewal asap.</p>
<p>Want to check, what domains, is it already tracking? Bhola comes with a very simple bare minimum UI, which one can use to check, what domains are being tracked, when are they expiring and other metadata details, like who issuer of the domain, when is it expiring, when was it issued.</p>
<p>What is required by Bhola to run? It just needs a good old postgres to function to keep track of the domains, and that’s it. Nothing shiny. Plain and simple.</p>
<p>Further on <a href="https://github.com/tasdikrahman/bhola/releases/tag/v0.2.0">v0.2</a> of bhola adds support for sending notifications to slack as part of evolving, from just being a dashboard to something which can be used to preemptively alert the operator on is the certificate expiring.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">It will alert for all the domains, which have already expired/are about to expire within the buffer period which you have set & send notification to your slack channel via webhook endpoint, periodically checking in the interval set by the operator, for expiration. (2/n) <a href="https://t.co/IdpDGxJQtr">pic.twitter.com/IdpDGxJQtr</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1312414866133512192?ref_src=twsrc%5Etfw">October 3, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Further more, smaller improvements like 1 step dev setup, docker-compose setup and container images available for docker would make reproducing the setup for bhola easier than before, than the status in milestone 0.1.</p>
<p>Not that it matters much, but <a href="https://rubyonrails.org/">rails</a> has been a fun framework to work on, especially with <a href="https://rspec.info/">rspec</a>, practicing <a href="https://en.wikipedia.org/wiki/Behavior-driven_development">BDD</a> and <a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a> has been a great experience.</p>
<h3 id="how-does-one-even-insert-a-domain-to-be-tracked-as-of-now">How does one even insert a domain to be tracked as of now?</h3>
<h5 id="example-request">Example request</h5>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl --location --request POST 'localhost:3000/api/v1/domains' \
--header 'Content-Type: application/json' \
--data-raw '{
"fqdn": "https://expired.badssl.com"
}'
</code></pre></div></div>
<h5 id="example-response">Example response</h5>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"data": {
"fqdn": "expired.badssl.com",
"certificate_expiring": true,
"certificate_issued_at": "2016-08-08T21:17:05.000Z",
"certificate_expiring_at": "2018-08-08T21:17:05.000Z",
"certificate_issuer": "/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Intermediate Certificate Authority"
},
"errors": []
}
</code></pre></div></div>
<h4 id="querying-the-domains-stored">querying the domains stored</h4>
<h5 id="example-request-1">Example request</h5>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl --location --request GET 'localhost:3000/api/v1/domains' \
--header 'Accept: application/json'
</code></pre></div></div>
<h5 id="example-response-1">Example response</h5>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"data": [
{
"fqdn": "www.tasdikrahman.com",
"certificate_expiring": false,
"certificate_issued_at": "2020-05-06T00:00:00.000Z",
"certificate_expiring_at": "2022-04-14T12:00:00.000Z",
"certificate_issuer": "/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA"
},
{
"fqdn": "expired.badssl.com",
"certificate_expiring": true,
"certificate_issued_at": "2016-08-08T21:17:05.000Z",
"certificate_expiring_at": "2018-08-08T21:17:05.000Z",
"certificate_issuer": "/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Intermediate Certificate Authority"
}
],
"errors": []
}
</code></pre></div></div>
<h3 id="are-there-other-ways-to-do-such-domain-expiration-checks">Are there other ways to do such domain expiration checks?</h3>
<p>Yes, there are are other ways to do this.</p>
<p>If you are on <a href="https://github.com/jetstack/cert-manager/">cert-manager</a>, then you can make use of <a href="https://grafana.com/grafana/dashboards/11001">https://grafana.com/grafana/dashboards/11001</a>, adding alerts on top of the dashboard should not be very complicated.</p>
<p>There is an exporter <a href="https://github.com/ribbybibby/ssl_exporter">https://github.com/ribbybibby/ssl_exporter</a>, which will do the scraping for you and expose the expiration date of the cert as the metric <code class="language-plaintext highlighter-rouge">ssl_cert_not_after</code>, which then you can add an alert on top.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>scrape_configs:
- job_name: "ssl"
metrics_path: /probe
static_configs:
- targets:
- example.com:443
- prometheus.io:443
</code></pre></div></div>
<p>Where <code class="language-plaintext highlighter-rouge">example.com</code> and <code class="language-plaintext highlighter-rouge">prometheus.io</code> would be your scrape endpoints, in this example.</p>
<p>Thanks to <a href="https://twitter.com/hashfyre/">Joy</a>, for pointing the above out to me.</p>
<p>Then there is a <a href="https://www.robustperception.io/get-alerted-before-your-ssl-certificates-expire">guide</a>, on how to do with prometheus, via the the <a href="https://github.com/prometheus/blackbox_exporter">blackbox exporter</a></p>
<p>And the good old <code class="language-plaintext highlighter-rouge">openssl</code> command will always be there</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">echo</span> | openssl s_client <span class="nt">-servername</span> www.tasdikrahman.com <span class="nt">-connect</span> www.tasdikrahman.com:443 2>/dev/null | openssl x509 <span class="nt">-noout</span> <span class="nt">-dates</span>
<span class="nv">notBefore</span><span class="o">=</span>Aug 19 13:59:14 2020 GMT
<span class="nv">notAfter</span><span class="o">=</span>Nov 17 13:59:14 2020 GMT
</code></pre></div></div>
<h3 id="how-does-me-running-bhola-help-then">How does me running bhola help then?</h3>
<p>The above tools will work, no doubt about it, if you already are on these systems, you can definitely leverage them to make use of them in ways similarly shown above. But even then, redundancy is never a bad thing to have.</p>
<p>Adding to it, one plus which bhola has is the validations before tracking endpoints, not tracking invalid/not having certs attached to endpoints. Which helps in keeping the entries sane.</p>
<p>If you are using <a href="https://letsencrypt.org/">letsencrypt</a>, given that the certificates would be expiring within <a href="https://community.letsencrypt.org/t/pros-and-cons-of-90-day-certificate-lifetimes/4621">3 months</a> and that they also send <a href="https://letsencrypt.org/docs/expiration-emails/">email notifications</a>, you would have used some automation to renew your certificates, due to the 3months expiration policy of LE certs. Having bhola as your external system to monitor your domains, would be an extra guard against the automation failing silently or the emails getting missed.</p>
<p>Furthermore, bhola panders more to the userbase, who are just in search of something, running which they can just start tracking and getting alerts for their domains, rather than tinkering with tools which they may be unfamiliar with, hence further reducing their friction in prioritizing their efforts to add alerting on domain expirations, rather than first trying to run <a href="https://github.com/prometheus/prometheus">prometheus</a>(if they aren’t already) or for those who don’t yet have the right level of automation maturity for their certificate renewals. If you/your org are already on a level where you have already done and dusted this alerting and cert renewal part via one of the ways described above or via some other way, then if I may say, you would come under a minority and not the norm.</p>
<p>Albeit there is an overhead with bhola which will be running a rails service and a postgres db. The decision then would with the operators, on what kind of solution would they be comfortable maintaining. People should always have multiple options for such tools to pick with, depending on their comfort level and the level of overhead they would want in their system.</p>
<h3 id="assumptions-made-by-bhola">Assumptions made by bhola</h3>
<ul>
<li>bhola assumes that the dns being inserted, resolves to a single IP, so in case you are doing dns loadbalancing on a single FQDN, with multiple IP’s behind it, it may try connecting to whichever IP first get’s returned.</li>
<li>bhola will not register the domain to be tracked, if it can’t reach it, it would be apt to place bhola somewhere, in your network, which would make it possible for bhola to resolve your dns endpoints with ease, so in case, the domains which you are trying to track, if they resolve to a private IP, make sure bhola can reach them.</li>
<li>bhola will not register the domain to be tracked, if it doesn’t have an SSL cert attached, it will not track it.</li>
</ul>
<h3 id="what-bhola-will-not-be">What bhola will not be</h3>
<ul>
<li>will not generate certificates for you by being the intermediate broker</li>
<li>will not install the certificates for it’s clients</li>
<li>will not provide a UI to generate/install/replace the certs for it’s clients</li>
</ul>
<h3 id="whats-next">What’s next?</h3>
<p>I envision bhola to be a 1 stop service for your needs of tracking your domain expirations for starters and there are quite a few thing which I want to see in it, in future. Some of them being</p>
<ul>
<li>Ability for it to associate domains and alerts with users, this will allow bhola to be multi-tenant.
<ul>
<li>The idea is to have a system in place if someone wants to enable this feature, this can be turned on with just enabling a feature flag when they start the
the webserver.</li>
<li>while accessing the api to do insertion for tracking domains, add authz/authn.</li>
</ul>
</li>
<li>Ability to delete domains associated with a user.</li>
<li>Ability to sign up using email id.</li>
<li>Ability to send alert notifications of all the domains associated to the user in their specified email id.</li>
<li>Not an immediate goal, but I want to host this on my own infrastructure, as a public facing endpoint.</li>
</ul>
<p>While I have not spread the above in specific milestones, but I would mostly pick up the first one for milestone 0.3.</p>
<p>As bhola is completely open source, would love to hear what you feel can be added to make <a href="https://github.com/tasdikrahman/bhola">bhola</a> better than before.</p>
Why I chose to do TDD for my new side project
2020-10-07T00:00:00+00:00
https://www.tasdikrahman.com/2020/10/07/why-I-chose-to-do-tdd-for-my-side-project
<p>This post is more of a continuation to this tweet</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">One thing which I tried doing differently this time with one of my side projects is to do TDD from the start. Someone may ask why? It's just a side project no? (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1312622418230296576?ref_src=twsrc%5Etfw">October 4, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>I have been building <a href="https://github.com/tasdikrahman/bhola">bhola</a> in my free time, and one thing which I tried doing differently this time with it, was to practice <a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a> from the start.</p>
<h2 id="but-why">But why?</h2>
<p>Someone may ask why? It’s just a side project no? True, yes. It is, but let me explain why I tried this out.</p>
<p>One reason is that, for some of my past side projects, when someone creates an issue/submits a PR. I wouldn’t necessarily remember everything which I did/why I did x instead of y, when I would have authored it (more on how this can be improved later)</p>
<p>Taking the liberty to quote <a href="https://twitter.com/AjeyGore">Ajey</a>.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">What ever code you write, it will be out of context in 18 months, write tests along with it, so at least people know what you meant</p>— Ajey Gore (@AjeyGore) <a href="https://twitter.com/AjeyGore/status/865555853423673344?ref_src=twsrc%5Etfw">May 19, 2017</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Coming back to say reviewing a bugfix/feature PR. Having no coverage for those specific routines which were modified, would mean I either would have to rely on my gut feeling, or I would have to test it by pulling the changes.</p>
<p>This in turn would do two things, for one, it would create a form of resistance, as to even review the PR, it would mean me having to also manually test out things and see if changes are not having any regression/the feature works as expected. Which would mean, I would either get swamped by the things to do to just review something, making the requests pile up one by one and then ending up in a position where there are multiple stale PR’s which have been just lying there. (If you have ever experienced this with any of my repositories, I sincerely apologise, I will strive to be better.)</p>
<p>The 2nd thing which would be a by-product of this, is that for these changes, I am doing the testing manually, which would mean I would have spent say x amount of time doing it which could have been used for something else.</p>
<p>This x amount of time, would vary wildly, depending on many factors. Some can be, how good are the docs, which would allow 1 to replicate the setup quickly(another reason why I really love 1 step dev setup commands)? Familiarity with the codebase so as to remember all the cases, corner cases included, so that you don’t miss them.</p>
<p>It’s natural for someone to not remember minute details of the codebase, when they are looking at it again after weeks/months/years. Naturally, they will need some time to again get acclimatized to the codebase which they had interacted/authored.</p>
<p>This is where tests for the routines bring in value. It’s your 1st level of safety net which you have spread out to weed out changes which would break your expected flow/behaviour.</p>
<h2 id="will-this-solve-all-my-problems">Will this solve all my problems?</h2>
<p>As luck will have it, I have an example from the side project which I was working on itself, where the coverage was high and covered the specific flow, but would ultimately fail when trying to run it!</p>
<p>There’s one specific flow, where the service reads an env var from the environment variable via <a href="https://github.com/laserlemon/figaro">Figaro</a>. The value is a plain boolean var of <code class="language-plaintext highlighter-rouge">true</code>. Now to simulate this in the spec, what I did was simply stub the call to the method of the figaro lib, to return the value I wanted for the flow. The problem being here, that I was stubbing the wrong type for the value! Figaro, when it reads the env var, it reads it as a string rather than a boolean(or any other type for that matter, all will be read as a plain string), which is where I was going wrong. This in turn would also affect the way, the implementation would happen.</p>
<p>Here’s a small snippet from the changelog of <a href="https://github.com/tasdikrahman/bhola/pull/65">https://github.com/tasdikrahman/bhola/pull/65</a> for reference, to give you an idea of what I am trying to depict here and what I changed to fix the same in the spec as well as the implementation.</p>
<div class="language-patch highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh">diff --git a/app/jobs/check_certificate_job.rb b/app/jobs/check_certificate_job.rb
index 93fb16b..76f995b 100644
</span><span class="gi">+++ b/app/jobs/check_certificate_job.rb
</span><span class="gd">--- a/app/jobs/check_certificate_job.rb
</span><span class="p">@@ -10,7 +10,7 @@</span> class CheckCertificateJob < ApplicationJob
Domain.all.each do |domain|
if domain.certificate_expiring?
Rails.logger.info("#{domain.fqdn} is expiring within the buffer period")
<span class="gi">+ if (Figaro.env.send_expiry_notifications_to_slack == 'true') && !Figaro.env.slack_webhook_url.empty?
</span><span class="gd">- if (Figaro.env.send_expiry_notifications_to_slack == true) && !Figaro.env.slack_webhook_url.empty?
</span> message = "Your #{domain.fqdn} is expiring at #{domain.certificate_expiring_not_before}, please renew your cert"
slack_notifier = SlackNotifier.new(Figaro.env.slack_webhook_url)
begin
<span class="gh">diff --git a/spec/jobs/check_certificate_job_spec.rb b/spec/jobs/check_certificate_job_spec.rb
index f4fa49b..71c40c8 100644
</span><span class="gi">+++ b/spec/jobs/check_certificate_job_spec.rb
</span><span class="gd">--- a/spec/jobs/check_certificate_job_spec.rb
</span><span class="p">@@ -48,7 +48,7 @@</span> RSpec.describe CheckCertificateJob, type: :job do
it 'will not call SlackNotifier#notify' do
allow_any_instance_of(Domain).to receive(:certificate_expiring?).and_return(true)
<span class="gi">+ allow(Figaro).to receive_message_chain(:env, :send_expiry_notifications_to_slack).and_return('false')
</span><span class="gd">- allow(Figaro).to receive_message_chain(:env, :send_expiry_notifications_to_slack).and_return(false)
</span> allow(Figaro).to receive_message_chain(:env, :slack_webhook_url).and_return(slack_webhook_url)
expect_any_instance_of(SlackNotifier).not_to receive(:notify).with(anything)
</code></pre></div></div>
<p>So as you see, it’s not necessary that following the above practices, will allow you to create bug free software.</p>
<p>Bhola had <a href="https://github.com/tasdikrahman/bhola/pull/65/checks?check_run_id=1203022679">~99.63%</a> coverage at the time this bug was present in it, but it didn’t stop it from having this bug.</p>
<p>100% code coverage doesn’t mean that your software is bug free/free of issues. The only real test is when your software is getting used by someone. This is where it should behave/perform as it is expected out of it. There’s <a href="https://en.wikipedia.org/wiki/No_Silver_Bullet">no silver bullet</a>.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Being proud of 100% test coverage is like being proud of reading every word in the newspaper. Some are more important than others.</p>— Kent Beck (@KentBeck) <a href="https://twitter.com/KentBeck/status/812703192437981184?ref_src=twsrc%5Etfw">December 24, 2016</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="so-whats-the-use-then">So what’s the use then?</h2>
<p>But having a high coverage would also mean, that you can refactor without fear, and have a faster feedback cycle than before, i.e testing for changes manually.</p>
<p>The 2nd level of safety net can be end-to-end integration tests for your codebase, which would run with each commit, the same way your unit tests would run with each commit.</p>
<p>The value here out of these 2 safety nets, is that you will be able to ship with more confidence, compared to not having these 2 safety nets at all</p>
<h2 id="why-i-chose-to-do-tdd-here">Why I chose to do TDD here?</h2>
<p>There’s a lot of literature around this, but for me personally I feel it allows me to think in terms of contract and how a routine should behave. As the behaviour is what we would really like to test for routine rather than the exact mechanics.</p>
<p>To add to it, the tests would act as documentation when I would go through them, telling me how a particular routine behaves under different scenarios. It also encourages baby steps and a faster feedback loop for something functional as fast as possible.</p>
<p>For reference, a few years ago, I wrote this thing called <a href="https://github.com/tasdikrahman/plino">plino</a>(spam filtering as an API) back in college days. I wasn’t aware of the testing literature back then (still learning), but what I ended up writing was an <a href="https://github.com/tasdikrahman/plino/blob/master/tests/test_plino_app_api_response.py">integration test</a> for the api.</p>
<p>It has absolutely no coverage for other routines which are present. It’s just by luck, that the codebase is small and someone will be able to quickly grok it and understand what is happening, but the overload of the same happening in larger codebase does affect maintenance.</p>
<p>If I have to compare it with bhola, I ended up having <a href="https://github.com/tasdikrahman/bhola/blob/master/spec/services/slack_notifier_spec.rb">coverage for even a small routine which just does a POST to an external API</a>. Someone might think it’s an overkill, why do we need all this if barely anyone is using this?</p>
<p>Another question which comes is, at the end it would be the functioning lines of code which your consumer of the software would be interacting with, not the tests. So why write tests? But would skipping these mean, taking a hit on maintainability, I feel the answer is yes.</p>
<p>As for <a href="https://github.com/tasdikrahman/bhola">bhola</a>, I feel I would definitely have more confidence and a faster feedback cycle when adding changes to it in future.</p>
<p>If you liked this piece, I have written a few more under <a href="https://www.tasdikrahman.com/blog/tag/testing">#testing</a> and <a href="https://www.tasdikrahman.com/blog/tag/tdd">#tdd</a>.</p>
Backpacking trip to Alleppey and Kochi
2020-08-31T00:00:00+00:00
https://www.tasdikrahman.com/2020/08/31/backpacking-trip-to-kerala-alleppey-fort-kochin
<blockquote>
<p>I did this trip last year, in August 2019. Finishing this was long overdue.</p>
</blockquote>
<p>Trying to follow up with the manager of <a href="https://www.tripadvisor.in/Restaurant_Review-g297628-d7925789-Reviews-Khawa_Karpo-Bengaluru_Bangalore_District_Karnataka.html">Khawa karpo</a> as Sushant and Rajat also finished their sharing of dinner which we were grabbing to call it a day. We all rushed as I grabbed my takeaway, to get a <a href="https://en.wikipedia.org/wiki/Rickshaw">rick</a> to catch my bus which was due to leave in about 15mins or so.</p>
<p>Didn’t try bargaining too much and just went ahead with the exorbitant charge for the meagre ~1km which was to be traversed.</p>
<p>Reaching the bus stop, Deepak was waiting already for me to board the bus whose tickets we had booked 2-3days ago.</p>
<p>Fast forward a few more tense moments, till we manage to switch seats with someone to have both of us sitting together, we settled into our seats for good and off we went, the bus zooming through the traffic as we edged closer to leaving the city behind. Zoning out looking out of the window, looking at the city dwellers leaving office and going back in their shiny cars and two wheelers made me realise that most of us are part of the same race, each day when we try getting back to out lives after work. Only difference today being, that I was escaping away back from the humdrum of the city to something new. That new being. God’s own Country - Kerala</p>
<p>As I devoured my takeaway from Khawa korpo, I felt giddy happy as I have always wanted to visit Kerala, which is something which I wanted to do while I was doing my <a href="https://www.tasdikrahman.com/2019/03/22/solo-backpacking-trip-to-hampi-gokarna-goa-budget/">solo trip</a> in the south-western edge some years back. And this was my moment I guess.</p>
<h2 id="alleppey">Alleppey</h2>
<p>And what better timing could it have been. I was visiting during the legendary <a href="https://en.wikipedia.org/wiki/Nehru_Trophy_Boat_Race">Nehru Trophy Boat Race</a>, the cherry on top being that <a href="https://en.wikipedia.org/wiki/Sachin_Tendulkar">Sachin Tendulkar</a> was gonna visit too being one of the chief guests.</p>
<p>Morning shined bright on our faces waking us up to the serene view in front of us, the drenched roads with the fresh bits of rainfall, the trees growing on the side of the road, making way for us as we swished passed them.</p>
<p>All of this, while checking google maps constantly to look for when do we need to get off the bus to be nearest to the hostel where we were gonna crash in Alleppey.</p>
<p>Similar to the gokarna trip, I managed to get bunk beds for me and Deepak in Zostel Alleppey, and off we went walking with our rucksacks towards the hostel, looking at google maps every now and then to check if we were not headed in the wrong direction.</p>
<p>Luckily, we got a rick to deliver us straight in front of the hostel. It was still super early for us to checkin to our rooms, but our host was kind enough to let us keep our rucksacks aside and let us use the washrooms.</p>
<p>I love how zostels have always ended up placing their hostels at some really gorgeous places. This being no different, it was right next to the Alleppey beach!</p>
<center><img src="/content/images/2020/07/kerala-blog-1.jpg" /></center>
<p>Another early bird was <a href="https://khatiwarasamiksha.wordpress.com/">Samiksha</a>, who had arrived maybe a day or two earlier than us to the zostel and we ended up striking a conversation with her and our host about the things which we had planned on doing.</p>
<p>As <a href="https://twitter.com/anirudh2403">Anirudh</a> had bailed out the last minute of the whole thing, we had an extra ticket spare with us for the Nehru tropy, which samiksha was ready to grab as we were laying down our plan for the day, to her.</p>
<p>After dilly dallying for a bit on whether to have breakfast right now or later, we went ahead with skipping it till we reached the venue as we heard from our host that the whole thing gets very crowded and the seats are assigned on a first come first serve basis, which made us not risk getting a distant seat in the stands.</p>
<p>As we arrived closer to the venue, famished with walking almost more than a km or two, guessing the main entrance of the race, which we had highly underestimated when we got down of our rick. We caved in with our hunger pang and decided to try out a small restaurant which was right next to the road which we had to follow to get to the entrance of the event.</p>
<p>To just describe this road, the road had an adjacent canal running next to it, which had a lot of boats, ranging from the size of a small canoe to that a decently sized ferry which would be able to carry at least 20-30 people. Their operators trying to woo us with their tarrifs, for them to chaperone one around the canals as well as the backwaters, bundling meals along with the whole deal.</p>
<p>Coming back to what we ate, I ordered some appam for myself, of which I repeated another serving of the same, along with some chicken dosa.</p>
<p>Filling out our tummies, gave us the energy to trod faster towards the venue and take our seats.</p>
<h3 id="nehru-trophy">Nehru Trophy</h3>
<center><img src="/content/images/2020/07/kerala-blog-2.jpg" /></center>
<p>The atmosphere was electric to say the least, everyone cheering their own team, clear evidence of a few rivalries between a few teams, the sound of the vuvuzela echoing the stands, the vendors shuttling around, trying to keep up with the demands of the buyers, rain droplets hitting your countenance as you cheer for the rowers from each team as they row with the mission of winning. All of this happening in the middle of the majestic backwaters.</p>
<center><img src="/content/images/2020/07/kerala-blog-3.jpg" /></center>
<p>It also accompanied a lovely cultural show being put up by folks as we started the race, which they were performing on the top of huge boats. Absolutely brilliant!</p>
<center><img src="/content/images/2020/07/kerala-blog-4.jpg" /></center>
<p>One thing which I was really happy seeing was that the organisers had asked for people to buy 10 rupee tokens to get stamped for their plastic bottles, and then they people can give the tokens back to the organisers by showing their number of bottles which they had gotten the token for to get back their token money, good way to incentivise people not throwing the plastic bottles.</p>
<p>We left at around lunch time as we passed the crowd off through the main gates, in our search for another restaurant where we would vanquish our thirst and pacify out tummies.</p>
<p>And the stubbornness to eat something authentic was really an itch we add had, so off we went in an aimless search for something which was cheap as well as something which was local.</p>
<p>We landed up at <a href="https://goo.gl/maps/spHQSjxY2AqXUTzs6">Subash Hotel</a>, not very far off from where we started searching for a hotel. And luck behold, it was also co-incidentally a <a href="https://en.wikipedia.org/wiki/Palm_wine">Toddy</a> (palm wine) shop. The host seeing all three of us a bit uncomfortable at first as we had samiksha with us, told us not to worry at all by showing us around the kitchen and his family who were helping him run the shop. The sight was a delightful one, where we could see freshly made tapioca, fried fish chips and of course toddy. We ended up settling down in one corner of the restaurant and out host helped us order a bit of everything which we saw in the kitchen. Me and Deepak didn’t have the Toddy, but samiksha managed to gulp down the whole Toddy bottle to herself. Our host sheepishly had a look at both me and Deepak as smiled at samiksha when both our glasses were empty and she was drinking straight out of the jar.</p>
<center><img src="/content/images/2020/07/kerala-blog-5.jpg" /></center>
<p>The road adjacent to the shop was also having a small canal running beside it, which gradually merged into Kollam-Kottapuram waterway and wasn’t it a sight to just look at the fresh green fields and the cocunut trees, we could see people occasionally drive(row) past us as we washed our hands after our fulfilling meal.</p>
<center><img src="/content/images/2020/07/kerala-blog-6.jpg" /></center>
<h3 id="alleppey-light-house">Alleppey Light house</h3>
<p>We ended up taking a rick back to our hostel as our legs had given up after all the running around since morning. As we zoomed past the roads, we also crossed the Alleppey beach light house, which has been since the time of erstwhile Travancore, before India’s independence. Alleppey light house catered to one of the busiest ports and trade centers in the southern coast, after the arrival of the dutch, portuguese and english traders.</p>
<p>Looking at this beauty, we got off at the light house as our hostel was not too far off, to look at the sun set from right next to the beach.</p>
<center><img src="/content/images/2020/07/kerala-blog-7.jpg" /></center>
<p>The beach was not very crowded and we ended up taking a small place right next to the pillars. Pillars? Yes, pillars. Or what was left off them. If you’re wondering what these pillars were doing here, my best guess would be that these were the remnants of the port where ships used to dock and load/unload the cargo. While the watchful lighthouse, gazed over all of us spraying light over the horizon as it moved it’s shower in circles.</p>
<p>We didn’t waste time in removing our shoes, keeping our socks aside and feeling the sand on our toes, the waves gushing in seaweed into our feet and us trying to keep up with not tumbling over into the sea by the shear strenght of the water gushing towards to us.</p>
<center><img src="/content/images/2020/07/kerala-blog-8.jpg" /></center>
<p>The pillars had rust from all the years of standing still, getting beat down by the waves and all saline water treating it all day long. Wild moss growing along it’s body, it almost felt like it was at this point part of the sea and an extension to it and not something mankind had put there.</p>
<p>We stayed around till the the sun set and all we that would guide us back were the shining stars above us, the hostel was barely a few hundred meters and off we went trodding down the beach with the occaional dog running around us.</p>
<p>Dog tired as we went back to our rooms, crashing back into our beds after taking a shower and quickly falling asleep. This marked the end of the first day for me and Deepak in Alleppey.</p>
<p>Me and Deepak had planned on visiting the beach early in the morning and that’s what we did after we woke up, we wore our sandals and off we went to just walk on the beach as the waves splashed onto the shore, taking back a bit of it with each visit it made.</p>
<center><img src="/content/images/2020/07/kerala-blog-9.jpg" /></center>
<p>The boats were parked in at the shores for the fisherman after they had come back from their hunt into the sea and the boats awaiter their masters as they stood there, their nose facing the sea, getting ready for their next shift.</p>
<p>While walking around, we encoutered bunch of puppies cuddled together to keep themselves warm with the perennial breeze of the sea. On coming closer, they all ran towards their mother who was probably in search of some food.</p>
<p>Which also reminded that we need to get some breakfast too! I didn’t order myself much, a sandwich and an omletter along with a serving of watermelon was all I had for breakfast as I was too occupied with planning on what we were going to do during the course of the day and as usual of me, we always made the plans on wwhich place to visit and what to do on the go. But not to digress, it has always been alright whenever I did so, which is why I wasn’t too worried, as long as we didn’t sit inside our rooms and not do anything.</p>
<p>Bugging our host on things to do, we managed to land ourselves a scooter for hire for the whole day. The catch was that Samiksha was to also accompany us along for the day, which would mean that either we had to land another scooter for rent from our host. Unluckily, the last scooter had been taken, but even if we got another scooter for ourselves, only Deepak amongst all three of us had an actual license for a 2 wheeler, hence even getting that extra scooter was not something which we could have had made much use. I know, I know. But I do have a license now. But not to digress, this meant that we had to do a tripple seater pillion ride wherever we were going.</p>
<h3 id="alleppey-coir-musueum">Alleppey Coir Musueum</h3>
<p>And that’s what we did, not very proud of it. But Deepak driving and both me and Samiksha, huddling together in the back with our raincoats, holding an umbrella, cutting through the wind, trying to save us all from getting more wet, off we went into towards the Coir museum, which was good ~8kms from our hostel. Luckily, it was a straight road and we covered the distance after a bit zooming around, trying to maintain our balance and not let the wind take away the umbrella.</p>
<center><img src="/content/images/2020/07/kerala-blog-10.jpg" /></center>
<p>The proctor of the hall was also kind enough to answer our questions about who used to work here and how many people would be working here on average on such machines. She also mentioned that the govt had incentivised studying coir courses and setup hostels for folks to come and learn the art while earning a small stipiend too which I felt was really helpful to attract more people to take this up while also sustaining themselves from the stipend money.</p>
<p>I had only heard about coir beds until that point, but I had no clue that one could do so many things using coir. We saw, actual beds, boats, houses, umbrellas, wall art, carpets, eco-friendly plant beds and what not. All of this being made from cocunut being the raw material.</p>
<p>Traditionaly coir was spun by bare hands, by simply twisting fibres between the palms of the hands, the introduction of ratts(spinning wheels) in the 19th century, significantly increased the productivity.</p>
<p>During the mid 18th century, a few europians with experience with jute products in Bengal, arrived in Alappuzha with two Bengali technicians to explore the prospects of the Coir yarn, on seeing their success, a lot of other industrialists set up their shops there. The establishment of such organised coir production facilities helped Alleppey become the unchallenged Coir Capital of the world.</p>
<h3 id="marari-beach">Marari beach</h3>
<p>The next stop was Marari beach, which was an odd ~7-8kms from the place which we were at. We could have gone with the usual way, which is the highway, but since we were doing a tripsy, we didn’t wanna take the risk of getting a ticket for this. Plus, what’s the fun on taking the highway when we had the alternative of going via the country side.</p>
<p>And this was probably one of the best decisions of the whole day. The roads were certainly very narrow at places, but the plus side of the whole thing was the fact that we could see everyone doing about their daily lives and us being the witness for it. Coir as we saw, was a huge industry, we saw a bunch of godowns which were operating out of small/mid-sized houses which from where they would probably process the fibre obtained from coconuts. The houses were built in a distinct style, where one could see the porch and the borders of the roof having a certain angle, which is very common to places where there is heavy rainfall, the columns were made of wooden, shaped like a cylinder, with the midriff being a bit broader from both the ends. And the verandah would usually be huge.</p>
<p>There came a point in the middle of all this, where we had to cross a huge puddle of water. Imagine 3 people holding to each other, trying to cross this puddle and not fall into it at the same time. Poor Deepak had to balance the scooter as well as both me and Samiksha, while we both were laughing it out while the poor chap took us across.</p>
<center><img src="/content/images/2020/07/kerala-blog-11.jpg" /></center>
<p>After circling around the beach for a bit (thanks to us for putting the wrong landmark), we finally reached the beach and it was serene. It has been the only beach, where I have actually noticed the sea cutting through land and delving inside the mainland. As the waves crashed on the shore, gnawing away at the shore, and taking parts and bits of whatever it could scavenge, inwards towards the mainland.</p>
<p>It was quite the sight.</p>
<center><img src="/content/images/2020/07/kerala-blog-12.jpg" /></center>
<p>The beach was luckily not that crowded when I reached there, plus also due to the fact that it was lunch time which added to the small number of people who were there.</p>
<center><img src="/content/images/2020/07/kerala-blog-13.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-14.jpg" /></center>
<p>After leaving Marari beach and it’s sand on our toes with us, we ended up in a restaurant which was on the side of the road, which we found out out of the blue. The food was amazing, although it was also the case that we had not eaten anything properly since the morning plus us being famished by the time we had left Marari. But not to take away from the restaurants food, it was quite good. On that note, we also managed to clean our feet and remove all the sand which we had gathered on our toes in the washroom of the restaurant after we ate.</p>
<h3 id="kumarakoram">Kumarakoram</h3>
<p>As usual, we did not have a clue on what to do next. We were sort of split between going for a boat house in the backwaters or going to the bird sanctuary which Samiksha was mentioning. At the end we decided to do whatever was open by the time we would have reached the place.</p>
<p>We took off on our scooty, to find someone who would be able to give us a quick boat ride. Yes, it was that random. We were literally scourging our way <a href="https://goo.gl/maps/ypftTyRqLFJfwqoa8">Aryakara</a>, trying to find for a party which would take us in their shelter, as it had also started drizzling quite hard. Imagine 3 people in an old beat down scooty, trying to just move around coast. This is when we stumbled upon a small narrow alley which was leading to the backwaters, we could also hazily see that there was a boat achored on the side and there were a few people standing there.</p>
<p>It started raining heavily and we had to park the scooter and hastily make a run for the shed. What we had stumbled upon was a govt outpost for jetty’s which would be used to transport people from one side to the other side of the backwater. We all looked at each other and decided, sure we can try this out, as it was not very far from the time when it would get dark, plus our search effort for a house boat was not leading us to anything special in particular, we decided to hop in into the next boat which was coming in, the fun part of it? We could also take our scooter inside with us as the trawler had provisions for it and so did the other few people who got in with us.</p>
<p>The tickets were subsidised and if I recall correctly, they costed less than 50INR individually, along with our scooter, which is pretty cheap if you ask me relatively. The next stop was Kumarakoram. As the trawler drifted past the sea plants, the view was breathtaking!</p>
<center><img src="/content/images/2020/07/kerala-blog-15.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-16.jpg" /></center>
<p>It must have been an odd 15-20mins by the time which we reached the coast of kumarakoram, the trawler took it’s own sweet time to reach the last end spot where we were supposed to get down. We could see the distance we had traversed, it was only trees only the swamp on our sides which gave way to the backwaters, at the horizon where we could see the fine line of trees of Aryakara which we had left behind.</p>
<center><img src="/content/images/2020/07/kerala-blog-17.jpg" /></center>
<p>On getting down from the boat and collecting our scooter, we headed straight to the small shops where we could see the owners of the houseboats, but sadly, the coversations were unweildy. We had already crossed the time until which they ply their services and they rightfully didn’t budge on that. Dejected, we decided to head back to our parked scooter and decided that we would take the scooter back all the way to our hostel.</p>
<p>Now we are talking about a good ~40kms give or take! In a scooter! While doing a trippsy!</p>
<p>But we had already made up our minds and off we went. It was certainly not comfortable before. Add on top of the fact that we were literally doing this from the time the day had started and we had embarked on our little adventure.</p>
<center><img src="/content/images/2020/07/kerala-blog-18.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-19.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-20.jpg" /></center>
<p>We crossed numerous boathouses, resorts, empty houses, fishermen coming back with their catch while on our way back our hostel. And when we finally reached, I just wanted to crash at my bed and even the hard cushion felt like an expensive high end mattress at the end of the day.</p>
<p>Just when the grasp of slumber was closing in, hearing my own grumbling tummy woke me back up from what little sleep that I was about to get. After freshening up, I pulled Deepak too outside of his room to sit in the common room.</p>
<p>It just so happened that a few folks ended up jamming together while we lazed away at the sidelines, our host and a few of his friends and some of the fellow hostelers</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/7z7F3QfI3MI" frameborder="0" allowfullscreen=""></iframe>
<p>I ended up singing while Samiksha did the strings, while Sakshi took was vlogging her trip video. Thanks Sakshi for capturing this. Do give a watch to her original <a href="https://www.youtube.com/watch?v=DbC14shIDxs">video</a> which I used to trim this part out.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/KcUNtQGxEio" frameborder="0" allowfullscreen=""></iframe>
<p>After all this, we went in search for a restaurant, at a time which I would easily consider quite late for given everything was more or less closed. On our way, we crossed a wedding ceremony, where we could see people dancing and being jolly. Now I have never done this before, but everyone more or less chimed in to agree upon us asking the folks inside if they had any food left! We literally ended up gatecrashing the thing and a few of us ended up dancing with the folks inside(Sakshi captures the same in her vlog)</p>
<center><img src="/content/images/2020/07/kerala-blog-21.jpg" /></center>
<p>There was a lot of banter and tomfollery which followed when the younger folks saw us with cameras or maybe they were just really having fun in the absense of the elders there to control what they were doing, either way, we let them all be and went ahead in search of a restaurant which we did find at the end of the whole thing(Sakshi has captured this too in her blog, alright, I will not repeat this again.)</p>
<p>We headed straight back to our hostel rooms and crashing for the night after the long day.</p>
<p>I ended up waking early in the morning and waking Deepak along with me too, to just run around the beach and spend sometime there, which we both had been planning to do. The best early morning beach view that I have ever had. We got to see small crabs, trying to dig through the sand, make a run for the sea as soon as they saw us. And just about that time, it also started raining quite heavily, we did not have any gear to protect us from the rain, which made us run towards the shelter to get some respite from the rain, as we watched the fishermen come back with their days catch. It was just beautiful, seeing the clouds pour their hearts out, flooding the sidelines of the porch of where we were standing. One thing which really stood out was a pair of swimmers, in full gear, cutting past the sea, further away from the shore, swimming against the waves, with each wave hitting them, their skill and practice clearly showing up as they moved away farther and farther away from the shore and we lost sight of them the moment they crossed the iron columns of the poles which used to be the erstwhile dock.</p>
<center><img src="/content/images/2020/07/kerala-blog-22.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-23.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-24.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-25.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-26.jpg" /></center>
<p>Did I forget to mention that, there was a local dog which had been accompanying us all the time since we had almost started from the hostel towards the light house?</p>
<p>This dog also managed to take away my flip-flops and make me run after itself for a good distance before he gave them up back to me. While the intension was clear for the dog that it wanted to play early in the morning for this, not so hostile human being which it found strolling around in the beach, it also managed to playfully bite (mouthing if you may) as we walked along the shore (more on this later.) much to our amusement.</p>
<p>Heading back to our hostel, we quickly took a shower and picked up our packed backpacks, heading for the train station before which we baded goodbye to Samiksha and whoever was awake when we were leaving.</p>
<p>We hadn’t eaten anything, which made us look for a decent place to fill our tummies, we did find a small eatery, which was being run by the railways. On settling down, we ordered a few dosas and cup of tea for each of us, while we waited for the train to arrive. The train did arrive, but little did we know that the train was to halt for only about 2 minutes. And here we were hastingly stuffing food into our mouths, trying to also eat and clear the bill, much to the amusement of everybody around us.</p>
<center><img src="/content/images/2020/07/kerala-blog-27.jpg" /></center>
<h3 id="kochi">Kochi</h3>
<p>The next leg in this trip was our visit to kochi, train tickets were pretty cheap, we got it for about less than 50INR each. The train itself was a passenger one, which meant it was a bit slower and halted at quite a few places, but that also gave us a chance to admire the countryside. The lush green meadows, the swamps and the local livestock grazing onto grass.</p>
<p>We reached in about an hour or two after we started to Ernakulam, from there we had to either take a ride back to the fort kochi or take a bus, we chose the latter. The only problem being, that we had a tough time finding a ride which would take us to fort kochi, with the clock ticking past 2pm and our heavy rucksacks behinds our backs, was making us rethink the whole idea of taking a bus ride to fort kochi but to rather just take a cab and just be done with it. But to our good luck, we did get a bus. which was also not very crowded.</p>
<p>The ride back to fort kochi was rougly around more than 10kms, which took us about another half an hour to reach. We saddled down our bags in a corner.</p>
<center><img src="/content/images/2020/07/kerala-blog-28.jpg" /></center>
<p>As luck would have it, our hostel where we were going to stay was hardly a few meters away from the bus stop which we got down off. And you might have guessed it right, it was a zostel again! We had to dump our backs in the common area, as the host was out for lunch, we quickly freshened up in one of the spare washrooms around the common area and off we went in search of some food.</p>
<p>We settled for this restaurant, right next to the fishing nets, which was a bit pricy? But well, we were really famished and we didn’t really mind paying premium for some decent food. For someone visiting, they should definitely try out the prawns and the likes when they visit.</p>
<center><img src="/content/images/2020/07/kerala-blog-29.jpg" /></center>
<p>It was a no brainer to decide on going for a walk alongside fort kochi and the fishing nets, once we were done with finishing up our meal.</p>
<center><img src="/content/images/2020/07/kerala-blog-30.jpg" /></center>
<p>If you haven’t seen the chinese fishing nets before, they are quite something to watch being operated by the fishermen, which is exactly what we did. They are huge, made with bamboo shoots and just about are the most complicated fishing nets I have ever seen, I am sure they are efficient than a lot more of the nets which we have around in Assam, where they use simple nets which you would imagine a fishermen to have, but this was entirely different which I hadn’t personally seen before. A couple of fishermen would be required to operate it.</p>
<p>Moving on, the boulevard which leads up to the edge of the fort, is bordered by a very old mansion. I say mansion, because you will have to actually see this once to gauge the sprawling lawn and experience the architecture of the house which is definitely back from the time Kochi was inhabited by the Portuguese and the Dutch. And there are a bunch of houses like this along the road and this not being a singular event.</p>
<center><img src="/content/images/2020/07/kerala-blog-31.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-32.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-33.jpg" /></center>
<p>We headed back straight to our hostel rooms after we had finished this small walk around the fishing nets, Deepak decided that he would sleep the afternoon out.</p>
<p>There was a <a href="https://en.wikipedia.org/wiki/Kathakali">Kathakali</a> which was about to start nearby the place where we were staying, called Kerala Kathakali centre, this was probably the first time I was going to watch this live with the performers doing their preparation live in front of the audience, this was definitely something which I didn’t want to miss out.</p>
<p>As we all settled down the small but cozy auditorium, the crowd was tring to find a seat as close as possible to the stage which would allow them to be closer to the whole spectacle.</p>
<center><img src="/content/images/2020/07/kerala-blog-34.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-35.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-36.jpg" /></center>
<p>The whole thing was just breathtaking, right from the expressions, to the preparation, to the dexterity and experience of the performers which made it look so fflawless. All in all, this is a great way in which they are preserving their culture and also showcasing it to the people of other cultures.</p>
<p>After this ended, I headed back to the hostel room, to finally give some rest to my feet, which was much required. Crashing on my bunk bed with the fan on top of my head, made me fall asleep for a good few hours, before being waken up by Deepak to head out for some dinner.</p>
<p>The next day, after waking up, we hired a scooter for rent for the whole day, which we planned to make full use of before we left from kochi, and off we went to get some breakfast. Next stop was <a href="https://www.tripadvisor.in/Restaurant_Review-g297633-d3738019-Reviews-Pepper_House_Cafe-Kochi_Cochin_Ernakulam_District_Kerala.html">Pepper house</a>, which is one of the few old spice houses back from the time when these buildings were used for actually storing spices, no surprises here on what pepper house used to store.</p>
<p>The view was lovely and the service was exemplery, although a bit pricey compared to other the eateries around, but would say definitely worth it.</p>
<center><img src="/content/images/2020/07/kerala-blog-37.jpg" /></center>
<p>After checking out the local art gallery and library inside pepper house, we headed out for the narrow streets of Mattencherry, which was also very close to Jew Town. The roads are filled with a lot of antique shops which sell everything right from really old antiques, paintings and souvenirs to collectibles which have been created by the localites.</p>
<center><img src="/content/images/2020/07/kerala-blog-38.jpg" /></center>
<center><img src="/content/images/2020/07/kerala-blog-39.jpg" /></center>
<p>The 16th century, Jewish Synagogue had a lot of rich culture and history associated with it, which was definitely fascinating. The best part of all this was the fact that, there were still a few local Jewish folks living around near the Synagogue.</p>
<p>We visited the naval museum after this, which had a bunch of historical writings around the naval history of Kochi, we also stumbled upon an officer back from Bangalore who asked us to tell more about the Ulsoor lake and the ASC center not from from it, such a small world. The funny part is, we got so lost in all the readings that somehow we did not notice that the musuem was closing down(the host at the gate, did tell us that it was supposed to close down at 5pm, but well we didn’t really keep track of time), and we almost got locked out inside the museum and had to literally rush back towards the entrance of the museum bunker (yes they have converted actual bunkers into musuems, how cool is that!)</p>
<p>After this, we went decided to head back to our hostel rooms, but before that, we headed to get some early dinner in one of the restaurants around the art galleries, around near the fishing nets, we didn’t fuss around too much and ate quickly, before heading back to drop off our scooter back to the garage from where we picked it up.</p>
<p>Reaching our hostel, we packed our stuff, and bade goodbye to the host, after hopping onto our taxi to head back to ernakulam Junction from where our train back to Bangalore was supposed to leave, trivia about this ticket was that, both of our tickets got confirmed a day before our departure.</p>
<p>Reaching bangalore early next morning, made us come back to our reality and realize that our trip was about to get/finally over.</p>
<p>I have very fond memories of Kerala. It was a lovely trip and I had a lot of fun writing this.</p>
<p>Until next time!</p>
A few notes on GKE kubernetes upgrades
2020-07-22T00:00:00+00:00
https://www.tasdikrahman.com/2020/07/22/a-few-notes-on-gke-kubernetes-upgrades
<blockquote>
<p>This post was originally published in <a href="https://blog.gojekengineering.com/how-we-upgrade-kubernetes-on-gke-91812978a055">Gojeks engineering blog, here</a>, this post is a cross post of the same</p>
</blockquote>
<p>This post is more of a continuation to this tweet</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">A few notes on <a href="https://twitter.com/kubernetes?ref_src=twsrc%5Etfw">@kubernetes</a> cluster upgrades on GKE (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1285619368353726465?ref_src=twsrc%5Etfw">July 21, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>If you are running kubernetes on <a href="https://cloud.google.com/kubernetes-engine">GKE</a>, chances are that you are already doing some form of upgrades for your kubernetes clusters, given that their release cycle is quarterly, which means you will have a minor version bump every quarter in the upstream. That is really a high velocity for version releases, but that’s not the focus of this post, the focus is on how you can attempt to keep up with this release cycle.</p>
<p>Quite a few things are GKE specific, but overall at the same time, there are also a lot of things which apply to in general any kubernetes cluster, whether self hosted or a managed one.</p>
<p>That being said. Let’s quickly set context, on what exactly do we mean by when we say a kubernetes cluster.</p>
<h1 id="components-of-a-kubernetes-cluster">Components of a kubernetes cluster</h1>
<p>In any kubernetes cluster, it would consist of your master and worker nodes. Two sets of nodes for different kind of workloads to run on them. More on the kind of workloads which run on them.</p>
<p>The master nodes in the case of GKE are managed by googlecloud
itself, now what does it entail? It means, the components like api-server, controller-manager, etcd, scheduler etc, will not have to be managed by you in this case, which would have been an added operational burden.</p>
<center><img src="/content/images/2020/07/components-of-kubernetes.png" /></center>
<p>I will not go into what each and every component goes into detail, as the docs do a good justice on what do they do, but just to summarise, the scheduler schedules your pods to nodes, the controller manager consists of a set of controllers used to control the existing state of the cluster and reconcile it with the state stored in etcd, the api-server is your entry point to the cluster, which is where each and every component comes to interact with other components.</p>
<h2 id="how-should-i-create-a-cluster">How should I create a cluster</h2>
<p>We use <a href="https://www.terraform.io/">terraform</a> along with gitops to manage the state of everything related to gcp, although I have heard good things about <a href="pulumi.com/">pulumi</a>, whatever works for you at the end of the day, but having the power of being able to declaratively configure the state of your infrastructure cannot be understated.</p>
<p>We have a bunch of cluster creation modules inside our private terraform repository, which makes creation of our GKE cluster, literally just a call to the module along with some sane defaults and the custom arguments which vary along with any cluster, a git commit and push, and then the next thing, which one sees is the terraform plan, right in the comfort of the CI, if all looks good, they do a terraform apply as the next step in the same pipeline stage.</p>
<p>With a few contextual things, about how we are managing the terraform state of the cluster, let’s move on to a few defaults which we set.</p>
<p>By default, one should always choose <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters">regional clusters</a>. The advantage of this is that, then GKE will maintain replicas of your control plane across zones, which makes your control plane resilient to zonal failures. Since the api-server is the entry to all the communication and interaction, this going down is basically you losing control/access to your cluster, but your workload will continue to run unaffected (if they don’t depend on the api-server or k8s control plane, more on this in some)</p>
<p>Components like Istio, prometheus operator, etc and even good old kubectl which depend on the api-server, may not function momentarily as the control-plane is getting upgrades in case your cluster is not a regional cluster.</p>
<p>Although, in the case of regional clusters, I haven’t personally seen any service degradation/downtime/latency increase while the master upgrades itself.</p>
<h2 id="master-upgrades-come-first-before-upgrading-anything">Master upgrades come first before upgrading anything</h2>
<p>The reason for that is, the control plane needs to be upgraded first and then the rest of the worker nodes.</p>
<p>When one is upgrading the master nodes(you will not see the nodes in GKE, but this would be running somewhere as part of VMs/borg pods et al. whatever google is using as an abstraction), the workloads running on it like the controller-manager, scheduler, etcd, the api-server are the components which are getting upgraded to the version of k8s which you are setting it to.</p>
<p>Master upgrades need to happen and then one can move on the worker node upgrades, the process of master node upgradation is very opaque in nature, as GKE manages the upgrade for you and not the cluster operator, which might not give you a lot of visibility on what exactly is happening. But nevertheless, if you just want to learn on what’s happening inside, you can try <a href="https://github.com/poseidon/typhoon">typhoon</a> and try upgrading the control plane of the cluster brought up using that, which I used to live upgrade the control plane of a self hosted k8s cluster in <a href="https://www.youtube.com/watch?v=3WgqFoo9eek&feature=youtu.be">devopsdays 2018 india’s talk</a>.</p>
<h2 id="gke-cluster-master-upgraded-what-next">GKE cluster master upgraded, what next</h2>
<p>The next obvious thing after you have done your GKE master node upgrade, is to upgrade your worker nodes, in the case of GKE, you would have node pools, which would in turn be having nodes being managed by the node pools.</p>
<p>Why different node pools? One can use separate node pools, to run different kind of node pools, which can then be used to segregate the workloads which run on the nodes of that node pool, for example. One nodepool can be tainted to run only prometheus pods, and then the prometheus deployment object can then tolerate that taint to get scheduled on that node.</p>
<h2 id="what-consists-of-the-worker-nodes">What consists of the worker nodes</h2>
<p>This is the part of the compute infra, which is what you get to interact with if you are on GKE.</p>
<p>These are the node pools, where your workloads will run at the end of the day.</p>
<p>As to components which make up the worker nodes, (excluding your workload) would be</p>
<ul>
<li>kube-proxy</li>
<li>container-runtime (docker for example)</li>
<li>kubelet</li>
</ul>
<p>Again, not going very deep into what each thing does, but on a very high level, kube-proxy is responsible for the translation of your service’s clusterIP to podIP at the end of the day, along with that also nodePort.</p>
<p>Kubelet is the process, which actually listens to the api-server for incoming instructions to schedule/delete pods to the node in which it is running. This instruction is in turn translated to the api instruction set which the container runtime (docker, podman for example) understands.</p>
<p>These 3 components, are again managed by GKE, and whenever you are upgrading your nodes, kube-proxy and kubelet gets upgraded, the container runtime need not receive and update while you upgrade. GKE would have it’s own mechanism to do it, but on a very high level, it does so by changing the image versions of the control plane pods.</p>
<p>We haven’t seen a downtime/service degradation happening due to these components getting upgraded on the cluster.</p>
<p>One good thing to note here is that, the worker nodes can run a few versions behind the version of the master nodes, the exact versions can be tested out on your staging clusters, to have more confidence while doing your production upgrade. But for example, I have seen if master is on 1.13.x, the nodes run just fine while even if they are on 1.11.x. Recommended is only a 2 minor version skew.</p>
<h2 id="anything-one-should-check-while-upgrading-to-a-certain-version">Anything one should check while upgrading to a certain version?</h2>
<p>Since the major <a href="https://github.com/kubernetes/kubernetes/releases">release cycle</a> is quarterly for kubernetes, one thing for sure which operators have to check is the release notes and the <a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/">changelog</a> for each version bump, as they usually entail quite a few api deletions and major changes.</p>
<h2 id="what-happens-if-my-cluster-is-regional-while-upgrading-it">What happens if my cluster is regional while upgrading it</h2>
<p>If your cluster is regional, the node upgrade happens zone by zone. You can control the number of nodes which can be upgraded at once using the surge configuration for the node pool, turning autoscaling off for the node pool is also recommended during the node upgrade upgrade.</p>
<p>If surge upgrade is enabled, a surge node with the upgraded version is created and it waits till the kubelet registers itself to the api-server, marking it ready after the kubelet reports the node as healthy back to the api-server, at which point, the api-server can direct the kubelet running on the surge node to schedule any workload pods</p>
<p>In case of a regional cluster, another node from the same zone is picked up to be drained, after which the node is cordoned, it’s workload rescheduled and then the end of this, the node gets deleted and removed from the nodepool.</p>
<h2 id="release-channels">Release channels</h2>
<p>Setting a <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels">release channel</a> is something which is highly recommended, we set it to stable for the production clusters, and the same for our integration clusters, with that being set, the nodes will always run the same version of kubernetes as the master nodes (excluding the small amount of time when the master is getting upgraded.)</p>
<p>There are 3 release channels, depending on how fast you want to keep up with the kubernetes versions released upstream</p>
<ul>
<li>rapid</li>
<li>regular (default)</li>
<li>stable</li>
</ul>
<p>Setting maintenance windows will allow you to control when these upgrade operations are supposed to kick in, once set, the cluster cannot be upgraded/downgraded manually, so if one doesn’t really care about granular control, they can not choose this option.</p>
<p>I haven’t personally downgraded a master version, please try this out on a staging cluster if you really need to. Although if you look at the docs, downgrading master is not really <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster#downgrading_limitations">recommended</a>.</p>
<p>Downgrading a node pool version is not possible, but you can always create a new node pool, with the said version of kubernetes and delete the older node pool.</p>
<h2 id="networking-gotchas-while-upgrading-to-a-version-114x-or-above">Networking gotchas while upgrading to a version 1.14.x or above</h2>
<p>If you are running a version lesser than 1.14.x and don’t have the ip-masq-agent running and if your destination address range falling under the CIDR’s</p>
<ul>
<li>10.0.0.0/8</li>
<li>172.16.0.0/12</li>
<li>192.168.0.0/16</li>
</ul>
<p>the packets in the egress traffic will not be masqueraded, which means that the node IP will be seen in this case.</p>
<p>The default behaviour after 1.14.x (and on <a href="https://cloud.google.com/container-optimized-os/docs">COS</a>), packets flowing from the pods stop getting NAT’d. This can cause disruption as you might not have whitelisted the pod address range.</p>
<p>One way is to add the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent">ip-masq-agent</a> and add the config for the nonMasqueradeCIDRs list the destination CIDR’s like 10.0.0.0/8 (for example if this is where your destination component like postgres lies), in this case the packets will use the podIP as the source address when the destination (postgres) receives the traffic and not the nodeIP.</p>
<h2 id="can-i-upgrade-multiple-node-pools-at-once">Can I upgrade multiple node pools at once?</h2>
<p>No you can’t, GKE doesn’t allow you to do so, even when you are upgrading one node pool, the node which get’s upgraded and picked for upgraded is not something which you will have control over.</p>
<p>So far we have discussed on what happens when the node pools get upgraded and the master node pools getting upgraded by GKE</p>
<h2 id="would-there-be-a-downtime-for-my-services-when-we-do-an-upgrade">Would there be a downtime for my services when we do an upgrade?</h2>
<p>Let’s start with the master component, if you have a regional cluster, while upgrading, since it happens zone by zone, even if your service is making use of the k8s api-server to do something, it will not get affected, although you should definitely try replicating the same for your staging setup assuming both have similar config.</p>
<p>Coming to the worker nodes. Well it depends.</p>
<h2 id="how-do-i-preventminimise-downtime-for-my-services-deployed">How do I prevent/minimise downtime for my services deployed?</h2>
<p>For stateless applications, the simplest thing to do is to increase the replicas to reflect the number of zones in which your nodes are present. But it need not to be necessary that scheduling of pods happen on each node across zone, kubernetes by default doesn’t handle this but gives you the primitives to handle this case.</p>
<p>If you want to distribute pods across zones, you can apply <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity">podantiaffinity</a> in the deployment spec for your service, with the topologyKey set to <code class="language-plaintext highlighter-rouge">http://failure-domain.beta.kubernetes.io/zone</code> for the scheduler to try scheduling it across zones. (You can read a more detailed post on scheduling rules which you can specify which I wrote sometime back <a href="https://www.tasdikrahman.com/2020/05/06/specifying-scheduling-rules-for-pods-with-podaffinity-and-podantiaffinity-on-kubernetes/">here</a>)</p>
<p>Distribution across zones will make your service resilient to zonal failures, the reason we increase the replicas to greater than 1 is that, when the nodes get upgraded, the node gets drained and cordoned and the pods get bumped out from that node.</p>
<p>In case the service which has only 1 replica, and it is scheduled on the node which is set for upgrade by GKE, whilst the scheduler tries finding itself a new node, there would be no other pod which is serving requests, which would cause a temporary downtime in this case</p>
<p>One thing to note here is that, if you are using <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/">PodDisruptionBudget(PDB)</a>, and if your running replicas are the same as the minAvailable specified in the PDB rule, the upgrade will just not happen, this is because of the fact that the node will not be able to drain the pod(s), as it respects the PDB budget in this case, hence you have to either</p>
<ul>
<li>increase the pods such that the running pods are > minAvailable specified in your PDB</li>
<li>remove the PDB rule specified.</li>
</ul>
<p>For <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">statefulSets</a>, you might have to take a small dowtime while you are upgrading, as the pods on which the stateful set is scheduled, that gets bumped out, the pvc claim will again be made by another pod, when it gets scheduled on the other node.</p>
<h2 id="but-tasdik-these-steps-of-upgrade-are-so-mundane">But Tasdik these steps of upgrade are so mundane</h2>
<p>Agreed, it is mundane, but there’s nothing stopping anyone from having a tool do these things for you, have a look at <a href="https://github.com/hellofresh/eks-rolling-update">eks-rolling-update</a>, GKE is way easier if you look at it. Which makes fewer touch points and cases where things can go wrong</p>
<ul>
<li>one being the pdb budget being the showstopper for your upgrade if you don’t pay attention</li>
<li>replicas being 1 or so for services</li>
<li>quite a few replicas being in pending or crashloopbackoff</li>
<li>statefulsets being an exception and needing to be handholded.</li>
</ul>
<p>For most of the above, you can initially start with having a fixed process (playbook) which one needs to follow and run through for each cluster whenever you are upgrading, so even though if the task is mundane, one knows which checks to follow and what to do to check the sanity of the cluster after the upgrade is done.</p>
<p>Replicas being set to 1 is just being plain naive, let your deployment tool have sane defaults of having replicas of 3 for minimum (3 zones in 1 region assuming you have podantiaffinity and a best case effort gets logged by the scheduler)</p>
<p>For the pods being in pending state, it means, you either are trying to request cpu/memory which is not available in any of the nodes present in the node pools, which again means, you are either not sizing your pods correctly, or there are a few deployments which are hogging resources, either way, it’s a smell that you are not having enough visibility into your cluster.</p>
<p>For statefulsets, I don’t think you can prevent not taking a downtime. So that’s there.</p>
<p>After all the upgrades are done, you can backfill the upgraded version numbers and other things back to your terraform config in your git repo.</p>
<p>Once you have rinsed and repeated these steps above, you can very well start with automating a few things.</p>
<h2 id="what-we-have-automated">What we have automated</h2>
<p>We have automated the part, where the whole analysis of what pods are running in the cluster, we extract this information out in an excel sheet. Information like</p>
<ul>
<li>replicas of the pods</li>
<li>age of the pod</li>
<li>status of the pods</li>
<li>which pods are in pending/crashloopbackoff</li>
<li>node cpu/mem utilization</li>
</ul>
<p>The same script handles inserting the team ownership details of the service, by querying our service registry and storing that info.</p>
<p>So all of the above details, at your tips, by just running the script in your command line and switching context to your clusters context.</p>
<p>As of now, the operations like</p>
<ul>
<li>upgrading the master nodes to a certain version</li>
<li>disabling surge upgrades/autoscaling the nodes</li>
<li>upgrading the node pool(s)</li>
<li>reenabling surge upgrades/autoscaling</li>
<li>setting maintenance window and release channel if already not set</li>
</ul>
<p>All the above being done via the CLI.</p>
<p>The next parts would be to automate the sequence in which these operations are done and codify the learnings and edge cases to the tool.</p>
<p>This is a bit mundane, no doubt. But this laundry has to be done, there’s no running away from it. Until one has automated the whole/major chunk of it, what we are currently doing in our team is to rotate people around doing cluster upgrades.</p>
<p>One person gets added to the roster, while there is a person who is already in the roster from the last week who will drive the upgrade for the week, also while giving context to the person who has just joined.</p>
<p>This helps in quick context sharing as well as the person who has just joined, they get to upgrade the clusters by following the playbooks, hence filling the gaps as we go forward.</p>
<p>The important part is that, you always come out of the week, with something improved, some automation added, some docs added. While also allocating dev time for automation explicitly in your sprint.</p>
<h2 id="ending-notes">Ending notes</h2>
<p>All in all, GKE really has a stable base which in turn allows us to focus more on building the platform on top of it rather than managing the underlying system and improve the developer productivity by building out tooling on top of the primitives k8s gives you.</p>
<p>If you compare this to something like running your own k8s cluster on top of VMs, there is a massive overhead of managing/upgrading/replacing components/nodes of your self managed cluster which in itself does require dedicated folks to handhold the cluster at times.</p>
<p>So if you really have the liberty, a managed solution is the way to go, take this from someone who has managed prod self hosted k8s clusters, and I will be honest, it’s definitely not easy, and something which if you can, should be delegated to focus on other problems</p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters">https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters</a></li>
<li><a href="https://github.com/kubernetes/kubernetes/releases">https://github.com/kubernetes/kubernetes/releases</a></li>
<li><a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/">https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/</a></li>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels">https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels</a></li>
<li><a href="https://cloud.google.com/container-optimized-os/docs">https://cloud.google.com/container-optimized-os/docs</a></li>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent">https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent</a></li>
<li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity">https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity</a></li>
<li><a href="https://www.tasdikrahman.com/2020/05/06/specifying-scheduling-rules-for-pods-with-podaffinity-and-podantiaffinity-on-kubernetes/">https://www.tasdikrahman.com/2020/05/06/specifying-scheduling-rules-for-pods-with-podaffinity-and-podantiaffinity-on-kubernetes/</a></li>
<li><a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/">https://kubernetes.io/docs/concepts/workloads/pods/disruptions/</a></li>
<li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/</a></li>
<li><a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/">https://kubernetes.io/docs/tasks/run-application/configure-pdb/</a></li>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster#downgrading_limitations">https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster#downgrading_limitations</a></li>
</ul>
<h2 id="credits">Credits</h2>
<ul>
<li>Image credits to <a href="https://en.m.wikipedia.org/wiki/File:Chrome_Vanadium_Adjustable_Wrench.jpg">wikipedia</a> and <a href="https://kubernetes.io">kubernetesio</a></li>
</ul>
Structured logging in Rails
2020-07-07T00:00:00+00:00
https://www.tasdikrahman.com/2020/07/07/structured-logging-in-rails
<blockquote>
<p>This post was originally published in <a href="https://blog.gojekengineering.com/structured-logging-in-rails-75e9a8c5370b">Gojeks engineering blog, here</a>, this post is a cross post of the same</p>
</blockquote>
<p>If you are on rails, you would have noticed that the rails logs which you get by default are quite verbose and spread across multiple lines, even if the context is of processing just one simple controller action.</p>
<p>What I will discuss in this post is how can one sanitize the logs, without losing out on information along with how you can add additional information for your log lines to make full use of the querying features of your logging platform.</p>
<p>What is not in scope of this blog is setting up the mechanism to push the logs from your rails app to your respective logging platform.</p>
<p>Let’s take the example of a simple health check controller which you would add to just make the health check pass for your rails app deployed on kubernetes</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/controllers/health_check_controller.rb</span>
<span class="k">class</span> <span class="nc">HealthCheckController</span> <span class="o"><</span> <span class="no">ActionController</span><span class="o">::</span><span class="no">Base</span>
<span class="k">def</span> <span class="nf">ping</span>
<span class="n">render</span> <span class="ss">json: </span><span class="p">{</span> <span class="ss">success: </span><span class="kp">true</span><span class="p">,</span> <span class="ss">errors: </span><span class="kp">nil</span><span class="p">,</span> <span class="ss">data: </span><span class="s1">'pong'</span> <span class="p">},</span> <span class="ss">status: :ok</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>and for the config (shown for the development evironment for this example)</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/environments/development.rb</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span>
<span class="n">config</span><span class="p">.</span><span class="nf">log_tags</span> <span class="o">=</span> <span class="p">[</span><span class="ss">:request_id</span><span class="p">]</span>
<span class="n">config</span><span class="p">.</span><span class="nf">log_level</span> <span class="o">=</span> <span class="ss">:debug</span>
<span class="k">end</span>
</code></pre></div></div>
<p>A simple route for the <code class="language-plaintext highlighter-rouge">GET</code> verb, to call the <code class="language-plaintext highlighter-rouge">#ping</code> action in the <code class="language-plaintext highlighter-rouge">HealthCheckController</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/routes.rb</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">routes</span><span class="p">.</span><span class="nf">draw</span> <span class="k">do</span>
<span class="n">get</span> <span class="s1">'/ping'</span><span class="p">,</span> <span class="ss">to: </span><span class="s1">'health_check#ping'</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This is what the logs would look like for the route <code class="language-plaintext highlighter-rouge">ping</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[my-app-fbf8d7bfc-wk5cd] I, [2020-07-01T07:19:05.007174 #1] INFO -- : [ec0ad0ba-dfb2-419b-bd81-5feb7dacb308] Processing by HealthCheckController#ping as HTML
[my-app-fbf8d7bfc-wk5cd] I, [2020-07-01T07:19:05.007874 #1] INFO -- : [ec0ad0ba-dfb2-419b-bd81-5feb7dacb308] Completed 200 OK in 0ms (Views: 0.2ms)
[my-app-fbf8d7bfc-wk5cd] I, [2020-07-01T07:19:05.290929 #1] INFO -- : [86332306-62e4-412e-a690-eee8253ab1c8] Started GET "/ping" for 10.177.3.1 at 2020-07-01 07:19:05 +0000
[my-app-fbf8d7bfc-wk5cd] I, [2020-07-01T07:19:05.292363 #1] INFO -- : [86332306-62e4-412e-a690-eee8253ab1c8] Processing by HealthCheckController#ping as HTML
</code></pre></div></div>
<p>You can notice a few things here,</p>
<ul>
<li>the logs are spread over multiple lines, adding to the difficulty in debugging the whole request/response flow.
<ul>
<li>given you would be pushing to your logging platform, let’s say EFK, which would allow you to do full text search, the configuration to have the request-id for each log would come handy. A little better than not having anything at all. (Have a look at <a href="https://github.com/BaritoLog">baritolog</a> if you haven’t, our in house EFK platform)</li>
<li>if you are not pushing to any logging platform, then you would be debugging this by tailing the logs of the rails app somewhere, if deployed to kubernetes, doing a <a href="https://github.com/johanhaleby/kubetail">kubetail</a> or if on VMs, sitting inside the VM and then tailing the logs of each and every application server instance. But then the first step here would be to obviously have centralized logging.</li>
</ul>
</li>
<li>extremely verbose, hindering debugging, if you don’t have that already.</li>
<li>no clear way to parse the logs as the logs format are non-standard (if you hadn’t added <code class="language-plaintext highlighter-rouge">config.log_tags = [:request_id]</code> which is not present by default)</li>
</ul>
<h2 id="ways-to-improve-this">Ways to improve this</h2>
<p>Taking things one at a time, you can compact out the log so that it stays meaningful and not verbose the way it is right now, by using something like <a href="https://github.com/roidrage/lograge">lograge</a>. There are a bunch of other alternatives like <a href="https://logger.rocketjob.io/">https://logger.rocketjob.io/</a> and <a href="https://github.com/shadabahmed/logstasher">https://github.com/shadabahmed/logstasher</a>, but I went ahead with lograge since it has been around for sometime now and has a larger adoption, my use case of what I required to do worked, and the use case for this as such was just to have certain things injected in our logs along with json formatted logging, and this worked very well for our use case.</p>
<p>Add lograge in your <code class="language-plaintext highlighter-rouge">Gemfile</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
gem 'lograge', '~> 0.11.2'
...
</code></pre></div></div>
<p>and do a <code class="language-plaintext highlighter-rouge">$ bundle</code>, after which you need to add setup the configuration as followed</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/environments/developement.rb</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span>
<span class="n">config</span><span class="p">.</span><span class="nf">lograge</span><span class="p">.</span><span class="nf">formatter</span> <span class="o">=</span> <span class="no">Lograge</span><span class="o">::</span><span class="no">Formatters</span><span class="o">::</span><span class="no">Json</span><span class="p">.</span><span class="nf">new</span>
<span class="n">config</span><span class="p">.</span><span class="nf">lograge</span><span class="p">.</span><span class="nf">enabled</span> <span class="o">=</span> <span class="kp">true</span>
<span class="n">config</span><span class="p">.</span><span class="nf">lograge</span><span class="p">.</span><span class="nf">base_controller_class</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'ActionController::Base'</span><span class="p">]</span>
<span class="n">config</span><span class="p">.</span><span class="nf">lograge</span><span class="p">.</span><span class="nf">custom_options</span> <span class="o">=</span> <span class="nb">lambda</span> <span class="k">do</span> <span class="o">|</span><span class="n">event</span><span class="o">|</span>
<span class="p">{</span>
<span class="ss">request_time: </span><span class="no">Time</span><span class="p">.</span><span class="nf">now</span><span class="p">,</span>
<span class="ss">application: </span><span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">class</span><span class="p">.</span><span class="nf">parent_name</span><span class="p">,</span>
<span class="ss">process_id: </span><span class="no">Process</span><span class="p">.</span><span class="nf">pid</span><span class="p">,</span>
<span class="ss">host: </span><span class="n">event</span><span class="p">.</span><span class="nf">payload</span><span class="p">[</span><span class="ss">:host</span><span class="p">],</span>
<span class="ss">remote_ip: </span><span class="n">event</span><span class="p">.</span><span class="nf">payload</span><span class="p">[</span><span class="ss">:remote_ip</span><span class="p">],</span>
<span class="ss">ip: </span><span class="n">event</span><span class="p">.</span><span class="nf">payload</span><span class="p">[</span><span class="ss">:ip</span><span class="p">],</span>
<span class="ss">x_forwarded_for: </span><span class="n">event</span><span class="p">.</span><span class="nf">payload</span><span class="p">[</span><span class="ss">:x_forwarded_for</span><span class="p">],</span>
<span class="ss">params: </span><span class="n">event</span><span class="p">.</span><span class="nf">payload</span><span class="p">[</span><span class="ss">:params</span><span class="p">].</span><span class="nf">except</span><span class="p">(</span><span class="o">*</span><span class="n">exceptions</span><span class="p">).</span><span class="nf">to_json</span><span class="p">,</span>
<span class="ss">rails_env: </span><span class="no">Rails</span><span class="p">.</span><span class="nf">env</span><span class="p">,</span>
<span class="ss">exception: </span><span class="n">event</span><span class="p">.</span><span class="nf">payload</span><span class="p">[</span><span class="ss">:exception</span><span class="p">]</span><span class="o">&</span><span class="p">.</span><span class="nf">first</span><span class="p">,</span>
<span class="ss">request_id: </span><span class="n">event</span><span class="p">.</span><span class="nf">payload</span><span class="p">[</span><span class="ss">:headers</span><span class="p">][</span><span class="s1">'action_dispatch.request_id'</span><span class="p">],</span>
<span class="p">}.</span><span class="nf">compact</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">config.lograge.base_controller_class = ['ActionController::Base']</code>, this part assumes that each controller will be inheriting from <code class="language-plaintext highlighter-rouge">ActionController::Base</code>, you can include any other controller which doesn’t inherit from the Base controller, in order for lograge to pick it up.</p>
<p>along with</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ApplicationController</span> <span class="o"><</span> <span class="no">ActionController</span><span class="o">::</span><span class="no">Base</span>
<span class="n">protect_from_forgery</span> <span class="ss">with: </span><span class="n">exception</span>
<span class="k">def</span> <span class="nf">append_info_to_payload</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="k">super</span>
<span class="n">payload</span><span class="p">[</span><span class="ss">:host</span><span class="p">]</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">host</span>
<span class="n">payload</span><span class="p">[</span><span class="ss">:remote_ip</span><span class="p">]</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">remote_ip</span>
<span class="n">payload</span><span class="p">[</span><span class="ss">:ip</span><span class="p">]</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">ip</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>now if you do a</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl -I localhost:3000/ping | grep -i "request"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
X-Request-Id: 4967a677-ab86-4a10-8a01-ea520951cf46
</code></pre></div></div>
<p>and check the logs of your rails app</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="nl">"level"</span><span class="p">:</span><span class="s2">"INFO"</span><span class="p">,</span><span class="nl">"progname"</span><span class="p">:</span><span class="kc">null</span><span class="p">,</span><span class="nl">"message"</span><span class="p">:</span><span class="s2">"{</span><span class="se">\"</span><span class="s2">method</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">GET</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">path</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">/ping</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">format</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">*/*</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">controller</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">HealthCheckController</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">action</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">ping</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">status</span><span class="se">\"</span><span class="s2">:200,</span><span class="se">\"</span><span class="s2">duration</span><span class="se">\"</span><span class="s2">:8.36,</span><span class="se">\"</span><span class="s2">view</span><span class="se">\"</span><span class="s2">:0.22,</span><span class="se">\"</span><span class="s2">db</span><span class="se">\"</span><span class="s2">:0.0,</span><span class="se">\"</span><span class="s2">request_time</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">2020-07-07 16:10:17 +0530</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">application</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">MyApplication</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">process_id</span><span class="se">\"</span><span class="s2">:7869,</span><span class="se">\"</span><span class="s2">params</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">{}</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">rails_env</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">development</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">request_id</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">4967a677-ab86-4a10-8a01-ea520951cf46</span><span class="se">\"</span><span class="s2">}"</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Now the whole log line captures a lot of metadata so that you can debug for each log line, you can also notice that one of the keys present in the log line is the <code class="language-plaintext highlighter-rouge">request_id</code> present, in this way, what is happening is, you have a unique trace id for your particular request, which you can do a full text search on your logging platform, if it supports it.</p>
<p>To extend this further, what one can do is capture the <code class="language-plaintext highlighter-rouge">request_id</code>, and pass it along the each controller’s flow, to capture it, you can simply do a <code class="language-plaintext highlighter-rouge">request.request_id</code>. Not that you have the <code class="language-plaintext highlighter-rouge">request_id</code>, if you are doing a <code class="language-plaintext highlighter-rouge">Rails.logger.{info|debug}</code> you can use it to log the <code class="language-plaintext highlighter-rouge">request_id</code>, this way, the request_id’s generated would now also be propagated to the custom logs which you would be adding to your application. The biggest advantage is, now you can add this <code class="language-plaintext highlighter-rouge">request_id</code> to each and <code class="language-plaintext highlighter-rouge">Rails.logger.{info|debug}</code> every core flow, which would be logged now, what this gives you is, you can just put the <code class="language-plaintext highlighter-rouge">request_id</code> in your search param in your logging platform, and it will give you all the logs which has this key in it.</p>
<p>You would also be able to capture additional information like host, remote_ip, ip of the rails app servicing it. by simply doing <code class="language-plaintext highlighter-rouge">request.host</code>, <code class="language-plaintext highlighter-rouge">request.remote_ip</code>, <code class="language-plaintext highlighter-rouge">request.ip</code>, <code class="language-plaintext highlighter-rouge">request.request_id</code></p>
<h3 id="why-not-use-lograge-for-even-the-custom-logger">Why not use lograge for even the custom logger?</h3>
<p>Lograge doesn’t support <a href="https://github.com/roidrage/lograge/issues/136">this</a>.</p>
<h3 id="what-should-i-do-now-to-add-structured-logging-for-my-custom-logs">What should I do now to add structured logging for my custom logs</h3>
<p>You can add a custom logger for your application and initialize it in your application config for the environments wherever you want it.</p>
<p>only thing you need to add is</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/logger/log_formatter.rb</span>
<span class="k">class</span> <span class="nc">LogFormatter</span> <span class="o"><</span> <span class="o">::</span><span class="no">Logger</span><span class="o">::</span><span class="no">Formatter</span>
<span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">severity</span><span class="p">,</span> <span class="n">time</span><span class="p">,</span> <span class="n">program_name</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="n">message</span> <span class="o">=</span> <span class="s1">''</span> <span class="k">if</span> <span class="n">message</span><span class="p">.</span><span class="nf">blank?</span>
<span class="n">severity</span> <span class="o">=</span> <span class="s1">''</span> <span class="k">if</span> <span class="n">message</span><span class="p">.</span><span class="nf">blank?</span>
<span class="p">{</span>
<span class="ss">level: </span><span class="n">severity</span><span class="p">,</span>
<span class="ss">progname: </span><span class="n">program_name</span><span class="p">,</span>
<span class="ss">message: </span><span class="n">message</span><span class="p">,</span>
<span class="p">}.</span><span class="nf">to_json</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\r\n</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>initialize it in the application config (development here in this case for this example)</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/environments/developement.rb</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span>
<span class="n">config</span><span class="p">.</span><span class="nf">log_formatter</span> <span class="o">=</span> <span class="no">LogFormatter</span><span class="p">.</span><span class="nf">new</span>
<span class="k">end</span>
</code></pre></div></div>
<p>and just to test the above out</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/controllers/health_check_controller.rb</span>
<span class="k">class</span> <span class="nc">HealthCheckController</span> <span class="o"><</span> <span class="no">ActionController</span><span class="o">::</span><span class="no">Base</span>
<span class="k">def</span> <span class="nf">ping</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="s2">"bazbar, request-id: </span><span class="si">#{</span><span class="n">request</span><span class="p">.</span><span class="nf">request_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">render</span> <span class="ss">json: </span><span class="p">{</span> <span class="ss">success: </span><span class="kp">true</span><span class="p">,</span> <span class="ss">errors: </span><span class="kp">nil</span><span class="p">,</span> <span class="ss">data: </span><span class="s1">'pong'</span> <span class="p">},</span> <span class="ss">status: :ok</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="how-will-my-logs-look-like-after-this">How will my logs look like after this?</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{"level":"INFO","progname":null,"message":"bazbar, request_id: 4967a677-ab86-4a10-8a01-ea520951cqw3"}
{"level":"INFO","progname":null,"message":"{\"method\":\"GET\",\"path\":\"/ping\",\"format\":\"*/*\",\"controller\":\"HealthCheckController\",\"action\":\"ping\",\"status\":200,\"duration\":8.36,\"view\":0.22,\"db\":0.0,\"request_time\":\"2020-07-07 16:10:17 +0530\",\"application\":\"MyApplication\",\"process_id\":7869,\"params\":\"{}\",\"rails_env\":\"development\",\"request_id\":\"4967a677-ab86-4a10-8a01-ea520951cqw3\"}"}
</code></pre></div></div>
<p>As you can see, the logs from both</p>
<ul>
<li>the controller logs which lograge is showing</li>
<li>the <code class="language-plaintext highlighter-rouge">Rails.logger.info</code> is spitting</li>
</ul>
<p>are having</p>
<ul>
<li>logs in json format</li>
<li>log line has the <code class="language-plaintext highlighter-rouge">request_id</code> appended to it in the message key</li>
</ul>
<p>That’s all folks</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://github.com/roidrage/lograge">https://github.com/roidrage/lograge</a></li>
<li><a href="https://www.paperplanes.de/2012/3/14/on-notifications-logsubscribers-and-bringing-sanity-to-rails-logging.html">https://www.paperplanes.de/2012/3/14/on-notifications-logsubscribers-and-bringing-sanity-to-rails-logging.html</a></li>
<li><a href="https://github.com/BaritoLog">https://github.com/BaritoLog</a></li>
<li><a href="https://stripe.com/blog/canonical-log-lines">https://stripe.com/blog/canonical-log-lines</a></li>
<li><a href="https://github.com/roidrage/lograge/issues/136">https://github.com/roidrage/lograge/issues/136</a></li>
<li><a href="https://medium.com/better-programming/ruby-on-rails-single-line-logging-5a76852de1d2/">https://medium.com/better-programming/ruby-on-rails-single-line-logging-5a76852de1d2/</a></li>
</ul>
<h2 id="credits">Credits</h2>
<ul>
<li>Image credits <a href="https://pixabay.com/photos/logs-timber-wood-logging-lumber-690888/">https://pixabay.com/photos/logs-timber-wood-logging-lumber-690888/</a> and <a href="https://en.wikipedia.org/wiki/Ruby_on_Rails">https://en.wikipedia.org/wiki/Ruby_on_Rails</a></li>
</ul>
Our learnings from Istio’s networking APIs while running it in production
2020-06-17T00:00:00+00:00
https://www.tasdikrahman.com/2020/06/17/our-learnings-from-istios-networking-apis-while-running-it-in-production
<blockquote>
<p>This was originally published under <a href="https://blog.gojekengineering.com/our-learnings-from-istios-networking-apis-while-running-it-in-production-74704979107d">Gojek’s engineering blog</a> by me, this post is a repost.</p>
</blockquote>
<p>We at Gojek have been running <a href="https://istio.io/">Istio</a> 1.4 with a multi-cluster setup for some time now, on top of which, we have been piloting a few reasonably high throughput services in production, serving customer-facing traffic.</p>
<blockquote>
<p>One of these services hits ~195k requests/minute.</p>
</blockquote>
<p>In this blog, we’ll deep dive into what we have learnt and observed by using Istio’s networking APIs.</p>
<h3 id="how-we-do-what-we-do">How we do what we do</h3>
<p>To help visualise the process better, let’s consider a workload that can be thought of as a logical unit (VMs, k8s pods, etc.), which is the source of traffic. A <strong>workload</strong> comprises of a service and an envoy proxy sidecar. Simply put, 2 workloads would comprise of 2 sets of service + proxy.</p>
<p>A <strong>service</strong> inside this workload is something present in the service registry, which is addressable over the network. Services define a name, which is typically a valid DNS hostname, a set of labeled network endpoints, ports, and protocols. The service registry could be the k8s service registry/consul, etc.</p>
<p>A <strong>gateway</strong> would be a proxy that receives traffic on specific ports, which can be a logical or a physical proxy in the network that defines L3-L6 behaviours, the sidecar (in this case, the Istio proxy) is also a gateway in that sense. Similar to workloads, a gateway also represents a source of traffic.</p>
<p>If you’re on a cloud provider, like GCP, you can create a service object of type Loadbalancer for the ingress gateway, and it’s not a bad idea to have staticIP for this.</p>
<p>With the above in picture, the networking APIs of Istio which we have dabbled with are <a href="https://istio.io/docs/reference/config/networking/virtual-service/">VirtualService</a>, <a href="https://istio.io/docs/reference/config/networking/destination-rule/">DestinationRule</a>, and <a href="https://istio.io/docs/reference/config/networking/gateway/">Gateway</a>.</p>
<h3 id="making-use-of-these-networking-apis">Making use of these networking APIs</h3>
<p>Here are a few combinations which can be used to achieve respective results:</p>
<ul>
<li>Ingress (Gateway + VirtualService)</li>
<li>Traffic splitting, TCP routing (VirtualService)</li>
<li>Canarying, Blue-green (VirtualService + DestinationRule)</li>
<li>Loadbalancing Config (DestinationRule)</li>
<li>Egress to external Services (ServiceEntry)</li>
</ul>
<p>One setup which we use is to create a default gateway for the cluster, which would handle all Ingress traffic for the services in the cluster. It’s also possible to have multiple gateways.</p>
<p>If one wants Ingress and nothing else, defining a VirtualService object for the service enables routing traffic to their service — Given the A record to point to the static IP of the gateway LB has been already created.</p>
<p>For capabilities like traffic shaping, an extra object of class DestinationRule has to be defined. This allows us to specify multiple subsets, which in turn need to be defined in the VirtualService spec to specify the weights for the different versions of the service. Based on the desired weights and rules defined in the VirtualService spec, the traffic needs to be routed.</p>
<h3 id="how-does-traffic-get-shaped">How does traffic get shaped?</h3>
<p>Traffic shaping happens in this form</p>
<center><img src="/content/images/2020/06/traffic-flow-1.png" /></center>
<p>although it may appear as below</p>
<center><img src="/content/images/2020/06/traffic-flow-2.png" /></center>
<p>When the traffic hits the gateway pods, the envoy proxy sidecar attached to the gateway would already know the pod IPs of the destination service. In simple words, the VirtualService and DestinationRules spec get translated to the envoy config.</p>
<h3 id="but-what-about-the-orchestration-of-crud-operations-of-these-resources">But… What about the orchestration of CRUD operations of these resources?</h3>
<p>Looking at this from an orchestration perspective, if one is to have 0 disruptions in the requests being serviced by the end service, the ordering of the creation of these resources also matter.</p>
<p>For starters, orchestrating the CRUD of these resources happens by putting them behind different CI pipelines and by using <a href="https://helm.sh/">Helm</a> would also work.</p>
<p>We use the <a href="https://github.com/istio/client-go">istio/client-go</a> to manage the lifecycle of the Istio resources, using our in-house deployment toolchain, powered by our service called Normandy (more on this in another blog post). This gets orchestrated by another service, which is our internal service registry, where we also store our service-related information and cluster mapping, along with other relevant information.</p>
<p>A big advantage of this approach of not letting a developer orchestrate the rules themselves is that we directly remove any room for errors that would occur because of the changes introduced manually. Leaving this to the tool in hand (Normandy and our service registry, in this case) allows us to put our logic inside a tool. This enables us to standardize the way we do the CRUD over the networking APIs of Istio, as well as the Kubernetes objects. Hence, any bugs we found would find their way back to the tool, standardizing, and enhancing the tool further.</p>
<p>If only VirtualService is being used to direct traffic to the service, the orchestration is pretty straight forward:</p>
<blockquote>
<p>Create the k8s service object and deployment object for the service, after which the VirtualService object is created. No changes to the gateway.</p>
</blockquote>
<p>The orchestration for deployment for a service would be:</p>
<blockquote>
<p>Deployment object → VirtualService update/create</p>
</blockquote>
<p>While using VirtualService + DestinationRule, the orchestration would be to:</p>
<blockquote>
<p>Do the CRUD of DestinationRule first, and then the CRUD of VirtualService. The above ordering is important.</p>
</blockquote>
<p>When there is already a VirtualService and DestinationRule existing for the service, and if the VirtualService is updated first, the subset being pointed to would not be present in the DestinationRule, causing 5xx errors till the DestinationRule is updated.</p>
<p>Everything seems in place now. There’s one subset in each VirtualService and DestinationRule for the service and things are working as usual without any 5xx errors while the orchestration is being done. But there are some unanswered questions.</p>
<blockquote>
<p>How does the above handle the case when there are more than 1 subsets, while doing some sort of traffic shaping, for instance? How is a zero-downtime deployment achieved?</p>
</blockquote>
<p>We didn’t have a use case with more than two subsets, as at any given point, we would only need to shape traffic to 2 versions of the service, but having more than 2 subsets is definitely possible.</p>
<p>Simply generating a new set of DestinationRule and VirtualService, and updating the existing one, in that order, is an approach that would work. However, this would cause timeouts for the service for a very brief period, and depending on the load, one might not see anything or encounter a 5xx error.</p>
<p>In a high throughput service, this definitely is a problem. More than that, the very process of a deployment causing 5xx’s feels wrong and seems like the system is not behaving the way it should. In such cases, it is the orchestration layer causing the problem.</p>
<p>Here’s an approach that would work:</p>
<p>1️⃣ orchestrateDestinationRule ⇨ 2️⃣ orchestrateVirtualService ⇨
3️⃣ cleanupDestinationRule</p>
<p>Step 1: Append on the newer subsets to the DestinationRule without removing the older subset. In this case, one will be appending the labels which distinguish their newer deployment object for the newer version of the service.</p>
<p>Step2: Update the VirtualService with the newer subset.</p>
<p>Step3: Clean up the subsets in the DestinationRule which are not required anymore.</p>
<p>Remember there’s a good chance one might hit the case of 5xx again even if the above steps have been followed, which is explained <a href="https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes">here</a>.</p>
<h3 id="why-does-the-5xx-arise">Why does the 5xx arise?</h3>
<p>A small fraction of time is required by the Envoy Proxy sidecars to get the updated DestinationRule and VirtualService config (in the case above) from pilot and synced on each proxy. Chances are, the envoy sidecar of the service pod might not have been updated yet when the requests come in. This in turn makes some of the proxy sidecars to have the correct updated config, while some still possess the old config.</p>
<h3 id="how-to-tackle-this">How to tackle this?</h3>
<p>A small hack would be to add a small sleep for ‘x’ amount of time while orchestrating, which would look like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>orchestrateDestinationRule ⇨ sleep(x) ⇨ OrchestrateVirtualService ⇨ sleep(x) ⇨ cleanupDestinationRule
</code></pre></div></div>
<h3 id="how-do-we-arrive-at-the-x-duration">How do we arrive at the ‘x’ duration?</h3>
<p>The metric <code class="language-plaintext highlighter-rouge">pilot_proxy_convergence_time</code> exposed can be tracked, to see how long it takes on an average, over a period of time, to infer the time it takes to sync the updated configs over each proxy sidecar. If the value appears to be on the higher side, try horizontally increasing the pilot replicas and check if that number comes down.</p>
<p>This is a hacky solution, but currently works on 1.4, as there is no other way to do this in the istio/client-go. We watch the status of the objects like DestinationRule and VirtualService, and check if they have been propagated everywhere in each proxy before moving on to orchestrating the next set of resources.</p>
<p>There is an option in the CLI to do a <code class="language-plaintext highlighter-rouge">istioctl experimental wait</code>, which blocks until the specified resource has been distributed.</p>
<p>For later versions of Istio, the Istio object status would have the information if the object has been reconciled with every proxy or not, which would help someone using the client-go to poll on this and proceed only if the resource has been propagated everywhere. This would be present in 1.6 directly, which also comes with a single binary for the control plane of Istio.</p>
<p>There is an issue tracking for this too. Find it <a href="https://github.com/istio/istio/issues/23956">here</a>.</p>
<p>Keeping the above discussion around orchestration in mind, the following cases would arise (accompanying the pseudocode) if you are:</p>
<ul>
<li>Trying to do normal deployments.</li>
<li>Doing Canary deployments with one version set as primary and the other version as candidate version. Note that in this case, as an identifier, we will use a deployment ID to distinguish between the two versions.</li>
</ul>
<p>For this example, we would assume that the service is named foo and below are two versions of DestinationRule and VirtualService spec for 2 subsets:</p>
<script src="https://gist.github.com/tasdikrahman/b30416bc8b6e521f016cd1209b5f5d43.js"></script>
<script src="https://gist.github.com/tasdikrahman/b20bb925b0a9977859dab861f13a2d38.js"></script>
<p><strong>Note</strong>: In the pseudocode ahead in the cases, the list object inside the DestinationRuleSpec and the VirtualService would represent the subset values, instead of having the whole spec presented, for the sake of simplicity of explanation.</p>
<h4 id="case-1">Case 1</h4>
<p>When it is a normal deployment, where the deployment, DestinationRule, and VirtualService for resource don’t exist.</p>
<h5 id="existing-resources">Existing resources:</h5>
<ul>
<li>Deployment Object: Not present</li>
<li>DestinationRule: Not present</li>
<li>VirtualService: Not present</li>
</ul>
<h5 id="orchestration-required-in-this-order">Orchestration required (in this order):</h5>
<ul>
<li>Create Deployment: foo-123</li>
<li>Create DestinationRule: [deployment-id: 123]</li>
<li>Create VirtualService: [subset: foo-123, weight: 100]</li>
</ul>
<h4 id="case-2">Case 2</h4>
<p>When it is a normal deployment, where the deployment, DestinationRule, and VirtualService for resource already exist.</p>
<h5 id="existing-resources-1">Existing resources:</h5>
<ul>
<li>Deployment Object: foo-123</li>
<li>DestinationRule: [deployment-id:123]</li>
<li>VirtualService: [subset: foo-123, weight: 100]</li>
</ul>
<h5 id="orchestration-required-in-this-order-1">Orchestration required (in this order):</h5>
<ul>
<li>Create — Deployment — foo-456</li>
<li>Update — DestinationRule — [deployment-id:123, deployment-id:456]</li>
<li>Update — VirtualService — [subset: 456, weight: 100]</li>
<li>Update — DestinationRule — [deployment-id: 456]</li>
<li>Delete — Deployment — foo-123</li>
</ul>
<h4 id="case-3">Case 3</h4>
<p>When it is a Canary deployment.</p>
<h5 id="existing-resources-2">Existing resources:</h5>
<ul>
<li>Deployment Object: foo-123</li>
<li>DestinationRule: [deployment-id: 123]</li>
<li>VirtualService: [subset: 123, weight: 100]</li>
</ul>
<h5 id="orchestration-required-in-this-order-2">Orchestration required (in this order):</h5>
<ul>
<li>Create — Deployment — foo-456</li>
<li>Update — DestinationRule — [deployment-id: 123, deployment-id: 456]</li>
<li>Update — VirtualService — [{subset: 123, weight: 90}, {subset: 456, weight: 10}]</li>
</ul>
<h4 id="case-4">Case 4</h4>
<p>When we are promoting a Canary deployment, with 456 being promoted.</p>
<h5 id="existing-resources-3">Existing resources:</h5>
<ul>
<li>Deployment — foo-123, foo-456</li>
<li>DestinationRule: [deployment-id:123, deployment-id:456]</li>
<li>VirtualService: [{subset: 123, weight: 90, subset: 456, weight: 10}]</li>
</ul>
<h5 id="orchestration-required-in-this-order-3">Orchestration required (in this order):</h5>
<ul>
<li>Update — VirtualService — [subset: 456, weight: 100}]</li>
<li>Update — DestinationRule — [deployment-id: 456]</li>
<li>Delete — Deployment — [foo-123]</li>
</ul>
<h4 id="case-5">Case 5</h4>
<p>When we are rolling back a Canary deployment, with 456 being rolled back.</p>
<h6 id="existing-resources-4">Existing resources:</h6>
<ul>
<li>Deployment — foo-123, foo-456</li>
<li>DestinationRule: [deployment-id: 123, deployment-id: 456]</li>
<li>VirtualService: [{subset: 123, weight: 90, subset: 456, weight: 10}]</li>
</ul>
<h5 id="orchestration-required-in-this-order-4">Orchestration required (in this order):</h5>
<ul>
<li>Update — VirtualService — [subset: 123, weight: 100]</li>
<li>Update — DestinationRule — [deployment-id:123]</li>
<li>Delete — Deployment — [foo-456]</li>
</ul>
<h4 id="case-6">Case 6</h4>
<p>When we are increasing the weight of Canary deployment, with 456 being rolled to a higher percentage (Example: 20%).</p>
<h5 id="existing-resources-5">Existing resources:</h5>
<ul>
<li>Deployment — foo-123, foo-456</li>
<li>DestinationRule: [deployment-id:123, deployment-id: 456]</li>
<li>VirtualService: [{subset: 123, weight: 90},{subset: 456, weight: 10}]</li>
</ul>
<h5 id="orchestration-required-in-this-order-5">Orchestration required (in this order):</h5>
<ul>
<li>Update: VirtualService: [{subset: 123, weight: 80}, {subset: 456, weight: 20}]</li>
</ul>
<p>One thing to point out above is that, after the CRUD operations of DestinationRule and VirtualService, one needs to add the wait time discussed above, in case v1.4 of Istio is being run. From v1.6, the status of the resource, whether it has reconciled on each and every proxy or not, can be polled directly on the resource object. More on this is described here.</p>
<p>This translates to do a poll on the <code class="language-plaintext highlighter-rouge">status.conditions[0].status</code> directly for the VirtualService and DestinationRule resource object, till we get the value True for the type <code class="language-plaintext highlighter-rouge">Reconcile</code>.</p>
<p>That’s all for now!</p>
<h3 id="references">References</h3>
<ul>
<li>The workload example is borrowed from the presentation given by Shriram Rajagopalan <a href="https://docs.google.com/presentation/d/1gtYNr8te2qeGgWrtH4S073JHmpbkKUkDHWvuLWTSenE/">here</a></li>
<li><a href="https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes">https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes</a></li>
<li><a href="https://github.com/istio/istio/issues/23956">https://github.com/istio/istio/issues/23956</a></li>
<li><a href="https://twitter.com/tasdikrahman/status/1267357405181968384">https://twitter.com/tasdikrahman/status/1267357405181968384</a></li>
<li><a href="https://istio.io/docs/reference/config/networking/virtual-service/">https://istio.io/docs/reference/config/networking/virtual-service/</a></li>
<li><a href="https://istio.io/docs/reference/config/networking/destination-rule/">https://istio.io/docs/reference/config/networking/destination-rule/</a></li>
<li><a href="https://istio.io/docs/reference/config/networking/gateway/">https://istio.io/docs/reference/config/networking/gateway/</a></li>
</ul>
AddTrust Root expiration fix
2020-05-31T00:00:00+00:00
https://www.tasdikrahman.com/2020/05/31/addtrust-root-expiration-fix
<p>With the root cert expiring for <a href="https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020">sectigo</a>, the older linux distributions are not properly ignoring the cert.</p>
<p>I have seen this affect boxes which ran ubuntu 16.04, but there would be others too. Didn’t notice anything on Debian 10(buster)</p>
<p>As people have pointed out <a href="https://www.reddit.com/r/linux/comments/gshh70/sectigo_root_ca_expiring_may_not_be_handled_well/">around</a>,
this is an openssl 1.0.2 bug. So even a system upgrade wouldn’t help the situation wouldn’t help, as this would require
an actual distro upgrade.</p>
<p>Programs which don’t depend on openssl(like go binaries), won’t get affected by this. Services/client on Ruby/Jruby for example, on the other hand will have problems similar to curl.</p>
<p>The same goes for programs which do certificate pinning on their clients. Personally, saw a saas vendor dole out a fix for this yesterday for their python client, These clients would see external calls to other endpoints failing.</p>
<p>This is also a great twitter thread by <a href="https://twitter.com/sleevi_/status/1266647545675210753">Ryan</a> on twitter</p>
<h2 id="how-curl-fails-in-a-typical-affected-client-box">How curl fails in a typical affected client box</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl https://myremoteserver.com
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
</code></pre></div></div>
<h2 id="what-should-i-do-if-i-am-affected-by-this">What should I do if I am affected by this?</h2>
<ul>
<li>the option to upgrade your distro is more or less out of the picture if you are currently affected by this, as you would need a tactical fix. This
being more of a long term strategic fix.</li>
<li><a href="https://www.agwa.name/blog/post/fixing_the_addtrust_root_expiration">Andrew</a> mentioned a fix which would involve putting a ! before mozilla/AddTrust Root cert</li>
<li><a href="https://twitter.com/kingslyj">kingsly</a> suggested this fix where we do a</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>echo -n > /usr/share/ca-certificates/mozilla/AddTrust_Low-Value_Services_Root.crt;\
echo -n > /usr/share/ca-certificates/mozilla/AddTrust_Public_Services_Root.crt; \
echo -n > /usr/share/ca-certificates/mozilla/AddTrust_Qualified_Certificates_Root.crt;\
echo -n > /usr/share/ca-certificates/mozilla/AddTrust_External_Root.crt;\
update-ca-certificates
</code></pre></div></div>
<p>The above would empty out the contents of the file, you could further use <a href="https://linux.die.net/man/1/chattr">chattr +i on the file</a> to change the file attributes such
that the above files are not modified from the state which we have set them to, but the bad part of the changing this attribute is that when do a system update,
this file would not get updated.</p>
<ul>
<li>A less disruptive approach, for debian based systems as pointed out by <a href="https://github.com/shanipribadi">shani</a> would be to do a</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo sed -i -e 's/^mozilla\/AddTrust_External_Root.crt/!mozilla\/AddTrust_External_Root.crt/' /etc/ca-certificates.conf
sudo update-ca-certificates --fresh
</code></pre></div></div>
Specifying scheduling rules for your pods on kubernetes
2020-05-06T00:00:00+00:00
https://www.tasdikrahman.com/2020/05/06/specifying-scheduling-rules-for-pods-with-podaffinity-and-podantiaffinity-on-kubernetes
<p>This is more of an extended version of the tweet here</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">If you haven't had a look at pod-affinity and anti-affinity, it's a great way which one can use to distribute the pods of their service across zones. <a href="https://t.co/iqhbyhruD8">https://t.co/iqhbyhruD8</a> (1/n)</p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/1231635358363729920?ref_src=twsrc%5Etfw">February 23, 2020</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>PodAntiAffinity/PodAffinity were released in beta some time back in 2017, in the <a href="https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes/">1.16 release</a> for k8s, along with node affinity/anti-affinity, taints and tolerations and custom scheduling.</p>
<p>Depending on your use case, you can definitely use a few of them for specific type of workloads, to achieve certain outcomes, for things running in your k8s cluster.</p>
<p>Will talk about how you can achieve distributing your pods across zones for your service using pod-affinity and anti-affinity.</p>
<p>One can use <code class="language-plaintext highlighter-rouge">preferredDuringSchedulingIgnoredDuringExecution</code>, which can be used for podAntiAffinity, for the scheduler to not be fixated on the constraints you put, rather it would give a best case effort to schedule the pods based on your constraints.</p>
<p>podAntiAffinity would use the pod labels to tell the scheduler, if pods of a particular label (eg: <code class="language-plaintext highlighter-rouge">app: foo</code>) are already scheduled on a node, then it would to schedule the pod in a node different than the one where the pod with above label is already scheduled.</p>
<p>The pods can be spread across zones by specifying <code class="language-plaintext highlighter-rouge">topologyKey: http://failure-domain.beta.kubernetes.io/zone</code> under <code class="language-plaintext highlighter-rouge">podAffinityTerm</code>, this assumes that the nodes are already spread across different zones to prevent catastrophe arising from zonal failure, having a regional cluster is a good idea</p>
<p>If you have multiple rules under <code class="language-plaintext highlighter-rouge">podAffinityTerm</code>, you can specify weights to them to tell the scheduler the importance of each rule, if you have just one rule, you can specify <code class="language-plaintext highlighter-rouge">100</code> to it.</p>
<p>A very simple example of podAntiaffinity being specified for a deployment, where we let the scheduler take a best case approach</p>
<script src="https://gist.github.com/tasdikrahman/b706b8e9c57d8b7ec3e311ab0dacb188.js"></script>
<p>You can notice, that the pods present here are getting scheduled in the same node, while the scheduler was trying it’s best to spread the pods out. If you had presumable 3 nodes, in 3 different zones, the rule above would have tried spreading out the pods in all the three zones, giving you resiliency over zone failures.</p>
<p>For the purpose of this example, I have a single node cluster in place.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ k get nodes
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready master 97s v1.14.10
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ k get pods -owide -l 'app=store'
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-cache-566bcff79-2qrzr 1/1 Running 0 110s 10.244.0.6 kind-control-plane <none> <none>
redis-cache-566bcff79-grhwk 1/1 Running 0 110s 10.244.0.7 kind-control-plane <none> <none>
redis-cache-566bcff79-ktmxv 1/1 Running 0 110s 10.244.0.5 kind-control-plane <none> <none>
</code></pre></div></div>
<p>One thing to note here is that, you can specificy multiple <code class="language-plaintext highlighter-rouge">matchExpressions</code> inside <code class="language-plaintext highlighter-rouge">podAffinityTerm.labelSelector</code>.</p>
<p>So if you wanted the scheduler to take into consideration another label like <code class="language-plaintext highlighter-rouge">app_type: server</code>, you can do something like</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- store
- key: app_type
operator: In
values:
- server
</code></pre></div></div>
<p>so both the match expressions will be <code class="language-plaintext highlighter-rouge">AND</code>‘d while the scheduler is evaluating the rules.</p>
<p>Now if we would want this rule to be enforced while being schduled you can use <code class="language-plaintext highlighter-rouge">requiredDuringSchedulingIgnoredDuringExecution</code> instead of <code class="language-plaintext highlighter-rouge">preferredDuringSchedulingIgnoredDuringExecution</code></p>
<script src="https://gist.github.com/tasdikrahman/927f44bf2efb1dacea51f316d4c53e42.js"></script>
<p>You would notice that the other two pods go into <code class="language-plaintext highlighter-rouge">Pending</code> state, since there is only one node present to spread out, given we are trying to enforce a required during scheduling rule, the scheduler would keep trying to schedule the other two pods to a node which doesn’t already have a pod with the label <code class="language-plaintext highlighter-rouge">app: store</code>, which isn’t able to find as there is only one node.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ k get pods -owide -l 'app=store'
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-cache-66b88fd4fc-7dc65 0/1 Pending 0 6s <none> <none> <none> <none>
redis-cache-66b88fd4fc-sbh5z 0/1 Pending 0 6s <none> <none> <none> <none>
redis-cache-66b88fd4fc-wszsq 1/1 Running 0 6s 10.244.0.8 kind-control-plane <none> <none>
</code></pre></div></div>
<p>Checking the logs of one of the pending pods would give the same</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 18s (x3 over 3m16s) default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules.
</code></pre></div></div>
<p>“IgnoredDuringExecution” means that the pod will still run if labels on a node change and affinity rules are no longer met. There were plans to have <code class="language-plaintext highlighter-rouge">requiredDuringSchedulingRequiredDuringExecution</code> which will evict pods from nodes as soon as they don’t satisfy the node affinity rule(s), but I haven’t seen that in the release docs, so not sure when is it scheduled to be added.</p>
<h3 id="why-not-use-node-anti-affinity">Why not use node anti-affinity?</h3>
<p>When node affinity/anti-affinity will be used to schedule pods in the specified nodes but it does not ensure the best effort to distribute pods evenly across nodes.</p>
<h3 id="side-effects-of-having-anti-affinity-rules">Side effects of having anti-affinity rules?</h3>
<p>Inter-pod affinity and anti-affinity require a substantial amount of processing which can slow down scheduling in large clusters significantly. The docs do not recommend using them in clusters larger than several hundred nodes.</p>
<p>Empty <code class="language-plaintext highlighter-rouge">topologyKey</code> is interpreted as “all topologies” (“all topologies” here is now limited to the combination of <code class="language-plaintext highlighter-rouge">http://kubernetes.io/hostname</code>, <code class="language-plaintext highlighter-rouge">http://failure-domain.beta.kubernetes.io/zone</code>, and <code class="language-plaintext highlighter-rouge">http://failure-domain.beta.kubernetes.io/region</code>). Each can be used to spread the pods using different constraints.</p>
<h3 id="how-is-it-different-from-taints">How is it different from taints?</h3>
<blockquote>
<p>Taints allow a Node to repel a set of Pods.</p>
</blockquote>
<p>So in order for a pod to be scheduled on a node, a pod has to <code class="language-plaintext highlighter-rouge">tolerate</code> that taint. This comes useful for workloads, where say you only want to schedule pods of a certain type on a set of nodes. A straight away use case is used in the control plane nodes of k8s, where you would not want your workload pods to get scheduled, or say nodes, where you only want to run <a href="https://prometheus.io/">prometheus</a> dedicatedly, given it can take quite a bit of memory.</p>
<h3 id="using-a-custom-scheduler">Using a custom scheduler</h3>
<p>Albiet I haven’t used it, but I feel it’s something which you should do only if you have a very good understanding of what’s gonna happen if you pluck out your default scheduler from you workloads podspec, as for in most cases, the default scheduler just works fine. Unless you are doing something really out of the usual.</p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity">https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity</a></li>
</ul>
A Few Notes on Etcd Maintenance
2020-04-24T00:00:00+00:00
https://www.tasdikrahman.com/2020/04/24/a-few-notes-on-etcd-maintenance
<blockquote>
<p>This was originally published under <a href="https://blog.gojekengineering.com/a-few-notes-on-etcd-maintenance-c06440011cbe">Gojek’s engineering blog</a> by me, this post is a repost.</p>
</blockquote>
<p>If you have worked around managing <a href="https://kubernetes.io/">Kubernetes</a> clusters on your infrastructure — instead of going with a managed version provided by cloud providers — chances are that you already are managing an <a href="https://etcd.io/">etcd</a> cluster. In case you are new to it, this post is for you.</p>
<p>We’ll get the basics out of the way first, and define what Etcd is.</p>
<blockquote>
<p>Etcd, is just a distributed key-value store, which uses <a href="https://raft.github.io/">raft</a> consensus algorithm in the back of it, to provide a fully replicated, highly available key-value store.</p>
</blockquote>
<p>A telling point of its stability is the fact that Kubernetes (API server) uses it as it’s key-value store for storing state of the whole cluster. It uses Etcd’s ‘watch’ function to monitor this data and to reconfigure itself when changes occur. The ‘watch’ function stores values representing the actual and ideal state of the cluster and can initiate a response when they diverge.</p>
<p>As to choosing Etcd over other databases like Redis, Zookeeper or Consul, I feel that it’s out of scope for this blog post considering there a lot of posts out there which go about detailing this. This post will try listing down a few things about the maintenance activities which we are currently running for our production Etcd databases, to keep them healthy.</p>
<h3 id="monitoring">Monitoring</h3>
<p>One of the first things which you can do is, adding a few basic alerts which will serve as the foundation for our maintenance activities.</p>
<script src="https://gist.github.com/tasdikrahman/26ac5d37a86a9360d4c361226308017b.js"></script>
<p>I will be only repeating myself here if I list down any more alert rules, as the official alert docs <a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/etcd3_alert.rules.yml">here</a> mention most of the important alerts. The one rule to notice apart from leader election and process being down is the DB size. This rule is not mentioned in the official rules doc, as this is something which people tune based on their DB size.</p>
<p>The metrics emitted by Etcd are in <a href="https://prometheus.io/">Prometheus</a> format. A few things on our side which we add while emitting is to add the environment label, depending on which environment the ETCD VM is running, as you can see in the alert above. You can remove that while trying it out on Alertmanager.</p>
<p>To avoid running out of space for writes, the keyspace history has to be compacted. Once reached, this would be obvious from the errors received by your client using Etcd.</p>
<p>Track <code class="language-plaintext highlighter-rouge">etcd_mvcc_db_total_size_in_bytes</code>, as etcd also emits <code class="language-plaintext highlighter-rouge">etcd_debugging_mvcc_db_total_size_in_bytes_gauge</code> in some versions. The reason to track the former, is that it can get dropped in later releases. It’s not a bad idea, to make sure to not track anything with <em>debugging</em> with it in the metric name</p>
<h3 id="compaction">Compaction</h3>
<p>As the space quota is limited, one would need to clear out the keyspace history, which can be achieved through compaction.</p>
<blockquote>
<p>Compaction truncates the per-key change log.</p>
</blockquote>
<p>Using one of these auto-compaction modes is usually recommended:</p>
<p>For periodic compactions, pass <code class="language-plaintext highlighter-rouge">--auto-compaction-retention</code> to the Etcd process while starting, eg: <code class="language-plaintext highlighter-rouge">--auto-compaction-retention=1</code> would run compaction every one hour. The mode picked up here would be periodic, this is similar to passing <code class="language-plaintext highlighter-rouge">--auto-compaction-mode=periodic</code></p>
<p>The other mode of compaction is revision-based, which is similar to passing <code class="language-plaintext highlighter-rouge">--auto-compaction-mode=revision</code>. We don’t use this as the use case for us is having a large keyspace rather than having a huge number of revisions for a key-value pair.</p>
<p>There’s a single revision counter which starts at 0 on etcd. Each change made to the keys and their values in the database is manifested as a new revision, created as the latest version of that key and with the incremented revision counter’s value associated with it.</p>
<p>For revision-based auto compaction mode, the behaviour should ideally be similar to when one runs something like <code class="language-plaintext highlighter-rouge">$ etcdctl compact 4</code>, after which the revisions prior to the compaction revision version become unavailable. So in this case, if you would do a <code class="language-plaintext highlighter-rouge">$ etcdctl get --rev=3 some-key</code> would fail in this case.</p>
<p>If you need more headspace, one can pass <code class="language-plaintext highlighter-rouge">--quota-backend-bytes</code> with the desired space to set space quota, while starting up the Etcd process. The default is 2GB, but you can max it up till <a href="https://www.ibm.com/support/knowledgecenter/SSBS6K_3.2.0/manage_cluster/manage_etcd_clusters.html">8GB</a>. It is ideal to keep this size lower.</p>
<h3 id="defragmentation">Defragmentation</h3>
<p>Compaction is not enough, as the internal DB exhibits fragmentation after compaction, leaving gaps in the backend database, which would still cause disk space to be consumed. This is fixed by running defragmentation on the Etcd DB.</p>
<blockquote>
<p>Defragmentation operation removes the free space holes from storage</p>
</blockquote>
<p>Combining compaction and defragmentation, along with the right set of monitoring can build the base for maintenance of your etcd cluster/node.</p>
<p>One thing to note here is that defragmentation should be run rather infrequently, as there is always going to be an unavoidable pause. Defragmentation to a live member blocks the system from reading and writing data while rebuilding its state.</p>
<p>This is it is recommended to keep the DB size smaller — the default again being 2GB. The larger the DB size being consumed, the more time it will take to defragment. Depending on how critical Etcd is to your app, you can take this into consideration.</p>
<h3 id="would-an-ha-setup-help-here">Would an HA setup help here?</h3>
<p>An <a href="https://en.wikipedia.org/wiki/High_availability">HA</a> Etcd setup won’t help in reducing the pause while defragmenting, as Etcd leader re-election will only happen when the defrag takes longer time than the leader election timeout, which shouldn’t happen if your DB is below 2GB and if compaction runs regularly.</p>
<p>In a clustered setup, defrag can be run per node, or can be directly run on the whole cluster by running <code class="language-plaintext highlighter-rouge">$ etcdctl defrag --cluster</code> to do it for all members of the cluster.</p>
<h3 id="running-defrag-operations">Running defrag operations</h3>
<p>Personally, I prefer a systemd.timer to run the defrag operation, compared to a traditional cronjob, as one would get the logs by default for each and every trigger by passing the timer service to journalctl, which is far more intuitive to someone logging onto the Etcd box.</p>
<script src="https://gist.github.com/tasdikrahman/d064d35d734d802e80a20e5138c839b9.js"></script>
<script src="https://gist.github.com/tasdikrahman/b450236232c048dd9aa0d1f173d7db18.js"></script>
<p>Using the above two systemd services, you can run the defrag operations based on when you would want it to run.</p>
<script src="https://gist.github.com/tasdikrahman/d2236b54c4d17f1df37cdfbdebc042a2.js"></script>
<h3 id="provisioning-etcd">Provisioning Etcd</h3>
<p>We use the <a href="https://github.com/chef-cookbooks/etcd/">Etcd community cookbook</a> to provision the Etcd servers, and haven’t noticed any issues so far.</p>
<script src="https://gist.github.com/tasdikrahman/6eccff66192b81e6f264391caa7bdb9f.js"></script>
<p>The sample systemd service is for a single node Etcd database. Depending on if you require an Etcd cluster or don’t want etcdv2 API to be available — among other things — the parameters to your <code class="language-plaintext highlighter-rouge">ExecStart</code> will change. (This is well-documented in the cookbook repo).</p>
<p>We haven’t tried running Etcd on Kubernetes. Although statefulSets are something slowly gaining traction, we didn’t feel running this on top of Kubernetes was something we wanted to do. That being said, we have heard good things about <a href="https://github.com/coreos/etcd-operator">etcd-operator</a>, although the project has been archived.</p>
<h3 id="so-how-do-we-automate-provision-of-etcd">So how do we automate provision of Etcd?</h3>
<p>We do it via a <a href="https://github.com/gojek/proctor">proctor script automation</a>, paired with a chef cookbook on top of <a href="https://github.com/chef-cookbooks/etcd">https://github.com/chef-cookbooks/etcd</a> with a few extra things, to create the defragmentation related systemd services.</p>
<h3 id="a-few-optimizations-you-can-do">A few optimizations you can do</h3>
<p>It is preferable to have the storage device attached to the Etcd VM to be in the same network, as Etcd is extremely dependent on storage latency, and some operations are blocked until most nodes have accepted a change. If latency passes certain critical numbers, one can end up with leader election storms, where no leader keeps the lease for long enough to actually make changes. It’s preferable to never use remote block storage for Etcd, as there are just too many places it could go wrong in weird ways.</p>
<p>Layering multiple layers of fault tolerance that aren’t aware of each other (in this case the remote storage) might lead you to end up with no fault tolerance at all.</p>
<p>Since Etcd is highly I/O dependant, it would make sense to have an SSD as the disk type attached to the ETCD instance, as this is something which is very critical to have lower disk write latency.</p>
<p>Sizing your VM type depending on your workload is crucial to not have performance issues (more discussion <a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/hardware.md">here</a>).</p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/maintenance.md">https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/maintenance.md</a></li>
<li><a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/etcd3_alert.rules.yml">https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/etcd3_alert.rules.yml</a></li>
<li><a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/hardware.md">https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/hardware.md</a></li>
<li><a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/etcd3_alert.rules.yml">https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/etcd3_alert.rules.yml</a></li>
<li><a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/grafana.json">https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/grafana.json</a></li>
<li><a href="https://freedesktop.org/software/systemd/man/systemd.timer.html">https://freedesktop.org/software/systemd/man/systemd.timer.html</a></li>
</ul>
<p>Thanks a ton to <a href="https://twitter.com/youngnick">@youngnick</a> for being super active on #etcd channel on Kubernetes Slack and helping us dig deeper, to <a href="https://twitter.com/Shubham5830">Shubham</a> for seeing this through with <a href="https://twitter.com/@tasdikrahman">me</a>, and my teammates for sticking with us.</p>
Introducing Kingsly — The Cert Manager
2020-04-22T00:00:00+00:00
https://www.tasdikrahman.com/2020/04/22/introducing-kingsly-the-cert-manager
<blockquote>
<p>This was originally published under <a href="https://blog.gojekengineering.com/introducing-kingsly-the-cert-manager-ced40746aa65">Gojek’s engineering blog</a> by me, this post is a repost.</p>
</blockquote>
<p>There’s one thing all devices connected to the Internet have in common — they rely on protocols called SSL/TLS to protect information in transit.</p>
<blockquote>
<p>SSL/TLS are cryptographic protocols designed to provide secure communication over insecure infrastructure.</p>
</blockquote>
<p>Any communication over the public internet should be encrypted, for which we need SSL certificates. There are many cases for public communication in GOJEK as well. Some of them are listed below:</p>
<ul>
<li>Client VPN connection</li>
<li>Inbound connections from mobile frontend to backend</li>
<li>Portals exposed over the public Internet.</li>
<li>Service to service communication over the public Internet</li>
</ul>
<p>While the industry is moving towards a <a href="https://cloud.google.com/beyondcorp/">zero trust network</a>, certificate management has been a big pain for us.</p>
<p>This post details how we built Kingsly, GOJEK’s open source certificate management tool.</p>
<h3 id="managing-ssltls-certs-until-yesterday">Managing SSL/TLS certs until yesterday</h3>
<blockquote>
<p>A certificate is a digital document that contains a public key, some information about the entity associated with it, and a digital signature from the certificate issuer.</p>
</blockquote>
<p>In other words, it’s a shell that allows us to exchange, store and use public keys. With that, certificates become the basic building block of the Public Key Infrastructure</p>
<p>We use letsencrypt heavily at GOJEK, to generate SSL/TLS certificates for numerous use cases, which can range from putting them behind our HAproxy Loadbalancers, envoy proxies, IPSec VPN’s and a lot of other places.</p>
<p>As of the end of 2018, the whole setup of renewing the certificates (it’s basically regenerating the certificate in the case of letsencrypt) installed at different places, was a manual process.</p>
<p>All letsencrypt SSL certificates, including renewals, are valid for no more than 90 days from their issue date. Thirty days before the certificate expires you will begin receiving renewal notices.</p>
<center><img src="/content/images/2020/04/kingsly_1.png" /></center>
<p>Seeing the above, someone from our team would renew these certificates manually and do the needful.</p>
<p>Although, this served our purpose for the time being, it was becoming difficult and time-consuming for one person to manage this piece of infrastructure with the increasing number of IPSec VPN’s.</p>
<p>Managing your own PKI infrastructure is a hard problem in itself and leaving it to manual processes always leaves room for human error.</p>
<p>The Kernel team, which I am part of, focuses on solving infrastructure problems, improving systems resiliency and developer productivity.</p>
<blockquote>
<p>Hence the motto: Productivity through Automation</p>
</blockquote>
<h3 id="problems-with-our-current-approach">Problems with our current approach</h3>
<p>We have faced a lot of issues around cert management:</p>
<ul>
<li>Using a wildcard certificate. If such certificate gets compromised, then all the subdomains are affected.</li>
<li>Renewal of expired certificates.</li>
<li>Certificate inventory. Keeping a list of hundreds of subdomains and maintaining expiry is an operational nightmare. Installation of letsencrypt locally on all the VMs makes it difficult to keep inventory at one central place.</li>
<li>Manual generation/non-standardized way of generating and renewal of SSL Certs leaves room of human error</li>
<li>Certs shared over email/slack.</li>
<li>No audit trail.</li>
<li>100+ Certs scattered across org with little or no visibility of when is it expiring until one gets an email.</li>
</ul>
<p>We need a solution to this problem — something very basic without a high learning curve.</p>
<h3 id="the-features-we-wanted-in-our-tool">The features we wanted in our tool</h3>
<ul>
<li>Certificates stored in a central manner</li>
<li>APIs exposed to create/renew the certificate</li>
<li>Automatic renewal a certain period before a cert expires</li>
<li>Centralised tracking and notification</li>
<li>Common API for internal users</li>
</ul>
<h3 id="enter-kingsly">Enter Kingsly</h3>
<p>It all started last December, when our team was contemplating what to ship for our internal company hackathon.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">It's going to be a looooong night here. 20+ teams taking part in our internal hackathon. <a href="https://twitter.com/gojektech?ref_src=twsrc%5Etfw">@gojektech</a> <br /><br />Let the games, sorry, hacks begin 😉 <a href="https://t.co/6yRjfo4GX6">pic.twitter.com/6yRjfo4GX6</a></p>— Gojek Tech (@gojektech) <a href="https://twitter.com/gojektech/status/1070276401885052928?ref_src=twsrc%5Etfw">December 5, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Our second internal hackathon in Bangalore starts today! 😎 <br /><br />We're pulling an all nighter starting at 4pm today to 12pm tomorrow noon. 😈<br /><br />01101000 01100001 01100011 01101011 <a href="https://t.co/BBN3Filu5D">pic.twitter.com/BBN3Filu5D</a></p>— Gojek Tech (@gojektech) <a href="https://twitter.com/gojektech/status/1070205884947750912?ref_src=twsrc%5Etfw">December 5, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>This tool was a perfect candidate for the night as it was both:</p>
<ul>
<li>a pain point</li>
<li>something which we really wanted to automate</li>
</ul>
<p>By the end of the night, we had it in a working condition and generating SSL certificates was as simple as doing a</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl -X POST http://kingsly.host/v1/cert_bundles \
-u admin:password \
-H 'Content-Type: application/json' \
-d '{
"top_level_domain":"your-domain.com",
"sub_domain": "your-sub-domain"
}'
</code></pre></div></div>
<p>and the response from the Kingsly server would be:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>'{
"private_key":"-----BEGIN RSA PRIVATE KEY-----\nFOO...\n-----END RSA PRIVATE KEY-----\n",
"full_chain":"-----BEGIN CERTIFICATE-----\nBAR...\n-----END CERTIFICATE-----\n"
}'
</code></pre></div></div>
<p>A simple JSON response which can be digested by a client the way it wants.</p>
<h3 id="we-had-a-framework-for-the-product-now-what">We had a framework for the product. Now what?</h3>
<p>We didn’t want this to end up as a hackathon project that remains a desolate creature in one’s source code repository. So we decided to extend Kingsly and make it into something which fit the initial set of requirements we had in mind for it.</p>
<h3 id="moar-features">Moar features!!</h3>
<p>After the cert generation part, the next feature which we wanted was automatic renewals of those certs.</p>
<p>The client (the IPSec client which we built) would keep polling the Kingsly server, get the certificates, and see if the current cert inside the IPSec VPN was different from the one which the server returned. If not, the client would replace the certs in the IPSec VPN box with the ones it received from the Kingsly server. It would then continue with the flow of tasks needed for the IPSec VPN to pick up the new certs, something which used to be done manually by a human.</p>
<center><img src="/content/images/2020/04/kingsly_2.png" /></center>
<h3 id="who-gets-to-request-generation-of-certs">Who gets to request generation of certs?</h3>
<p>This was still an unsolved problem in the whole equation. We would need to devise a way in which we would be able to deny requests from clients.</p>
<p>One thing which we were sure about was: authentication and authorization should not be handled by the application.</p>
<p>A very simple solution was to put the Kingsly web server behind an HAproxy. Here, we would have a frontend rule pointing to a backend having an ACL with a list of IPs which would be allowed through.</p>
<p>This checked the initial problem of only allowing the clients which were whitelisted.</p>
<script src="https://gist.github.com/tasdikrahman/e65c36f8156eeef855c5fc24196655bb.js"></script>
<p>But the other problem would be maintaining the updated list of all the clients being allowed. On top of that, we were trying to automate a process. Hence, settling for something which eventually needed manual intervention didn’t quite fit our vision. So, we started exploring other possibilities and stumbled upon IAP, which fit the bill for our use case.</p>
<blockquote>
<p>Identity Aware Proxy (IAP) is the GCP provided solution for user as well as service authentication.</p>
</blockquote>
<p>IAP is nothing but Oauth 2.0 implementation over a proxy.</p>
<p>For authentication with IAP, some required headers need to be added to every request which goes out of the client box. We created a proxy service, which adds the authentication details to the request. For service-to-service authentication, it uses OAuth 2.0 using creds of service accounts.</p>
<center><img src="/content/images/2020/04/kingsly_3.png" /></center>
<h3 id="why-use-iap-for-auth">Why use IAP for auth?</h3>
<ul>
<li>Central authorization layer</li>
<li>Application level access control</li>
<li>Allows individual and group-based access policies</li>
<li>Enforces HTTPs</li>
<li>Mandatorily redirects HTTP requests to HTTPS</li>
<li>Attaches client identity to request headers for further processing by downstream applications/proxy</li>
</ul>
<h3 id="theres-more-to-come">There’s more to come</h3>
<ul>
<li>Build support for HAproxy, envoy proxy and the other places where we require x.509 certificates which will be interacting with Kingsly, in the kingsly-certbot client.</li>
<li>Have better monitoring to ensure whether certificates installed correctly.</li>
<li>Allow authorization on top of Kingsly, to open it up to developers to enable them to request SSL certificates and usher the road towards SSL/TLS enabled applications.</li>
<li>CRD to generate certs for applications inside our k8s clusters. We will be evaluating the use cases similar to how we did for Kingsly before going ahead with it.</li>
</ul>
<p>As the saying goes, there’s no silver bullet in software and Kingsly checks off the list of requirements we had from a tool.</p>
<blockquote>
<p>Kingsly is open source! Check the links below 🖖</p>
</blockquote>
<h3 id="links">Links</h3>
<ul>
<li><a href="https://github.com/gojekfarm/kingsly">https://github.com/gojekfarm/kingsly</a> — the API server which handles cert generation and renewals.</li>
<li><a href="https://github.com/gojekfarm/kingsly/tree/master/docs/deploy/k8s">https://github.com/gojekfarm/kingsly/tree/master/docs/deploy/k8s</a> — kingsly-server and worker deployment docs</li>
<li><a href="https://github.com/gojekfarm/kingsly-certbot">https://github.com/gojekfarm/kingsly-certbot</a> — the IPSec VPN client for kingsly</li>
<li><a href="https://github.com/gojekfarm/kingsly-certbot-cookbook">https://github.com/gojekfarm/kingsly-certbot-cookbook</a> — chef cookbook to setup certbot while provisioning an IPSec VPN.</li>
<li><a href="https://github.com/gojekfarm/iap_auth">https://github.com/gojekfarm/iap_auth</a> — Proxy for talking to an IAP enabled service.</li>
<li><a href="https://github.com/gojekfarm/iap_authenticator">https://github.com/gojekfarm/iap_authenticator</a> — Ruby Gem to create authentication token for services running behind IAP</li>
<li><a href="https://github.com/gojekfarm/iap-auth-cookbook">https://github.com/gojekfarm/iap-auth-cookbook</a> — chef cookbook to setup up the iap-auth proxy binary on an IPSec VPN box.</li>
</ul>
<p>Kingsly was also presented by me over at DevConf, India</p>
<script async="" class="speakerdeck-embed" data-id="3e0b36f62907438094dde34369646e50" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
Route missing in kubernetes node with kuberouter as the CNI
2020-01-05T00:00:00+00:00
https://www.tasdikrahman.com/2020/01/05/routes-missing-kubernetes-kube-router-networkd
<p>Anyone who is evaluating into having a networking solution for their kubernetes cluster without having a lot of moving parts in the cluster, <a href="https://www.kube-router.io/">kuberouter</a> provides <a href="https://kubernetes.io/docs/concepts/cluster-administration/networking/">pod networking</a>, ability to enforce <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/">network policies</a>, IPVS/LVS service proxy among other things.</p>
<p>The problem which we faced specifically while running this in our clusters was missing routes upon restart of the node, or sometimes in the case when the node was joining the cluster as part of the worker node.</p>
<p>For us, the issue would come around as a the <a href="https://github.com/uswitch/kiam">kiam</a> (which we were using for identity management for pods inside the k8s clusters) pod would go into <a href="https://stackoverflow.com/questions/44702715/kubernetes-pod-fails-with-crashloopbackoff">CrashLoopBackOff</a> as described by me in the github issue <a href="https://github.com/uswitch/kiam/issues/49">https://github.com/uswitch/kiam/issues/49</a>, as the dns resolution would fail (more on that later)</p>
<p>We were using the latest version of <a href="http://coreos.com/">Coreos</a>, but we found out that the version 1576.5.0 of Coreos was not plagued by this problem.</p>
<p>This has been defined in detail in the <a href="https://github.com/cloudnativelabs/kube-router/issues/370">github issue</a></p>
<p>The problem was that there was race condition caused by <a href="https://www.freedesktop.org/software/systemd/man/systemd-networkd.service.html">systemd-networkd.service</a> trying to manager the tunnels and modigying the routes causing the missing routes. Whenever networkd was restarting, all the tunnels would go away with it.</p>
<p>It is best described by <a href="https://github.com/cloudnativelabs/kube-router/issues/370#issuecomment-399850110">Niel here</a>.</p>
<p>To fix this, a file in the networkd dir <code class="language-plaintext highlighter-rouge">/etc/systemd/network/</code> so it starts ignoring those interfaces and doesn’t manage them as described by <a href="https://github.com/cloudnativelabs/kube-router/issues/370#issuecomment-463967949">Lomkju</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Match]
Name=tun* kube-bridge kube-dummy-if
[Link]
Unmanaged=yes
</code></pre></div></div>
<p>It was tested to be working for the following coreos version as mentioned by <a href="https://github.com/cloudnativelabs/kube-router/issues/370#issuecomment-463967949">Lomkju</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Container Linux by CoreOS 1967.6.0 (Rhyolite)
Kernel: 4.14.96-coreos-r1
</code></pre></div></div>
Various ways of enabling canary deployments in kubernetes
2019-09-12T00:00:00+00:00
https://www.tasdikrahman.com/2019/09/12/ways-to-do-canary-deployments-kubernetes-traefik-istio-linkerd
<p><strong>Update</strong> I gave a quick lightening talk about the same talk @ DevopsDays India, 2019. The slides for which can be found below</p>
<script async="" class="speakerdeck-embed" data-id="0b063af43dd54ea794077749a347b58e" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<h3 id="what-canary-can-be">What canary can be</h3>
<p>Shaping the traffic in a way, so that we could direct a % of traffic to the new pods and promoting the same deployment to a full scaleout and gradually phasing out the older release.</p>
<h3 id="why-canary">Why canary?</h3>
<p>Testing on staging doesn’t weed out all the possible reasons for something failing, final testing for a feature being done on some part of the traffic is not something unheard of.
Canary being a precursor to enable full blue green deployments.</p>
<h4 id="why">Why?</h4>
<p>If you don’t use feature flags for your services, canary testing becomes paramount to test out features.</p>
<h4 id="approaches-to-enable-canary">Approaches to enable canary</h4>
<h5 id="using-bare-bone-deployments-in-k8s">Using Bare bone deployments in k8s</h5>
<h6 id="how">How?</h6>
<p>Creation of two sets of deployments and referring to figure 1, v1 and v2, both would be separate deployment objects with separate label selectors. Both v1 and v2 deployments would be exposed via the same svc object which would point to their pods.</p>
<h6 id="advantages">Advantages</h6>
<ul>
<li>Plain and simple.</li>
<li>Can be done without any plugins/extra stuff in the vanilla k8s we get in GKE.</li>
</ul>
<h6 id="disadvantages">Disadvantages</h6>
<p>Traffic will be a function of replicas, and cannot be customized. For example, if the traffic splitting is done between the two deployment with v1 having 3 replicas and v2 having 1 replica, the traffic split for canary will be 25%</p>
<h5 id="using-istio">Using Istio</h5>
<h6 id="how-1">How?</h6>
<p>In an istio enabled cluster, we need to set the routing rules to configure the traffic distribution.</p>
<p>Similar to the approach above, we have two deployments and svc objects for the same service, called v1 and v2.</p>
<p>The rule will look something like</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> - <span class="o"><<</span><span class="no">EOF</span><span class="sh">
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: helloworld
spec:
hosts:
- helloworld
http:
- route:
- destination:
host: helloworld
subset: v1
weight: 90
- destination:
host: helloworld
subset: v2
weight: 10
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: helloworld
spec:
host: helloworld
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
</span><span class="no">EOF
</span></code></pre></div></div>
<h6 id="advantages-1">Advantages</h6>
<ul>
<li>Has been out there in the wild for quite some time. i.e Battle tested</li>
<li><a href="https://medium.com/google-cloud/automated-canary-deployments-with-flagger-and-istio-ac747827f9d1">Flagger</a> can be added to have automated canary promotion.</li>
<li>GKE has an add on feature which can be used to install istio in our clusters</li>
<li>Traffic routing and replica deployment are two completely orthogonal independent functions</li>
<li>Focused canary testing, eg: instead of exposing the canary to an arbitrary number of users, if you wanted the users from some-company-name.com to the canary version, leaving the other users unaffected, you can do that too with a rule to match the headers for the match to check for the cookie for the above.</li>
</ul>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
spec:
hosts:
- helloworld
http:
- match:
- headers:
cookie:
regex: <span class="s2">"^(.*?;)?(email=[^;]*@some-company-name.com)(;.*)?$"</span>
...
</code></pre></div></div>
<ul>
<li>Tracing gets as a side benefit</li>
</ul>
<h6 id="disadvantages-1">Disadvantages</h6>
<ul>
<li>Another add on to manage inside the cluster if gone through the route of installing the istio version</li>
</ul>
<h5 id="using-linkerd">Using Linkerd</h5>
<h6 id="how-2">How?</h6>
<p>Linkerd has a canary CRD that enabled how a rollout should occur</p>
<p>It automatically creates two sets of deployments for a deployment name <code class="language-plaintext highlighter-rouge">podinfo</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
podinfo ClusterIP 10.7.252.86 <none> 9898/TCP 96m
podinfo-canary ClusterIP 10.7.245.17 <none> 9898/TCP 23m
podinfo-primary ClusterIP 10.7.249.63 <none> 9898/TCP 23m
</code></pre></div></div>
<p>And then you have to do a rollout to start the traffic shaping to happen to the <code class="language-plaintext highlighter-rouge">podinfo-canary</code>.
A detailed post on how to do this is <a href="https://linkerd.io/2/tasks/canary-release/">here</a></p>
<h6 id="advantages-2">Advantages</h6>
<ul>
<li>Flagger can be integrated for automated canary promotion/demotion</li>
<li>Battle tested and has been in use by the virtue of being the first service mesh
<h6 id="disadvantages-2">Disadvantages</h6>
</li>
<li>Another component inside the k8s cluster to be maintained</li>
</ul>
<h5 id="using-traefik">Using Traefik</h5>
<h6 id="how-3">How?</h6>
<p>It requires the same setup of two deployment and svc objects for the service which needs to have canary enabled for it and makes use of the ingress object in k8s to define the traffic split between the services.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
traefik.ingress.kubernetes.io/service-weights: |
my-app: 99%
my-app-canary: 1%
name: my-app
spec:
rules:
- http:
paths:
- backend:
serviceName: my-app
servicePort: 80
path: /
- backend:
serviceName: my-app-canary
servicePort: 80
path: /
</code></pre></div></div>
<h6 id="advantages-3">Advantages</h6>
<ul>
<li>Easy to setup with no frills, as it’s just an ingress controller in the k8s controller (like contour/ nginx-ingress-controller)</li>
<li>Doesn’t need the pods to be scaled.</li>
<li>Support for tracing included.</li>
</ul>
<h6 id="disadvantages-3">Disadvantages</h6>
<ul>
<li>No inbuilt process to shift weights from v1 to v2 or revert back traffic in case of increased error rates. Ie. not a clear cut way to integrate with flagger for automated canary promotion/demotion</li>
</ul>
<h4 id="references">References</h4>
<ul>
<li><a href="https://martinfowler.com/bliki/CanaryRelease.html">https://martinfowler.com/bliki/CanaryRelease.html</a></li>
<li><a href="https://martinfowler.com/bliki/FeatureToggle.html">https://martinfowler.com/bliki/FeatureToggle.html</a></li>
</ul>
<p>Canary using istio</p>
<ul>
<li><a href="https://istio.io/blog/2017/0.1-canary/">https://istio.io/blog/2017/0.1-canary/</a></li>
</ul>
<p>Bare bones canary on k8s</p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments">https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments</a></li>
</ul>
<p>Canary using traefik</p>
<ul>
<li><a href="https://blog.containo.us/canary-releases-with-traefik-on-gke-at-holidaycheck-d3c0928f1e02?gi=c58435e35526">https://blog.containo.us/canary-releases-with-traefik-on-gke-at-holidaycheck-d3c0928f1e02?gi=c58435e35526</a></li>
<li><a href="https://www.tasdikrahman.com/2018/10/25/canary-deployments-on-AWS-and-kubernetes-using-traefik/">https://www.tasdikrahman.com/2018/10/25/canary-deployments-on-AWS-and-kubernetes-using-traefik/</a></li>
<li><a href="https://docs.traefik.io/user-guide/kubernetes/#traffic-splitting">https://docs.traefik.io/user-guide/kubernetes/#traffic-splitting</a></li>
</ul>
<p>Canary using linkerd</p>
<ul>
<li><a href="https://linkerd.io/2/tasks/canary-release/">https://linkerd.io/2/tasks/canary-release/</a></li>
<li><a href="https://linkerd.io/2/features/traffic-split/">https://linkerd.io/2/features/traffic-split/</a> Done using https://flagger.app/</li>
<li><a href="https://www.tarunpothulapati.com/posts/traffic-splitting-linkerd/">https://www.tarunpothulapati.com/posts/traffic-splitting-linkerd/</a>
Excerpt: “Flagger combines traffic shifting and L7 metrics to do canary deployments, etc. It will slowly increase the weight to the newer version, based on the metrics, and if there is any problem (e.g. failed requests), it would roll back. If not it will continue increasing the weight until all the requests are routed to the newer version. Tools like Flagger can be built on top of SMI and they work on all the meshes that implement it.”</li>
</ul>
Handling signals for applications running in kubernetes
2019-04-24T00:00:00+00:00
https://www.tasdikrahman.com/2019/04/24/handling-singals-for-applications-in-kubernetes-docker
<p>When the power goes off in a device in a linux based system, one can think of ways in which this event can be handled in the applications running on it. One thing to note is that, when you plug the power cable off, the power doesn’t really go off immediately.</p>
<p>But this needs to be notified to processes so that they can handle such an event and save the state of the application (if any).</p>
<p>A few possible ways of doing so would be to</p>
<ul>
<li>Save everything in RAM to disk and then restore the contents stored on disk back to RAM when the startup happens next, but the problem with this approach is that, storing and retrieval on/from a disk is a slow process.</li>
<li>Use a file to store the state of power, where 0 would denote that the power is on and 1 would mean that the power has gone off, but this approach would mean that the processes running in the system should keep track of this bit stored in the file.</li>
<li>The kernel sends a signal like <code class="language-plaintext highlighter-rouge">SIGTERM</code> to the process and leaves it to the process on how it handles this signal.</li>
</ul>
<p>How a container gets stopped is something which is very important or not, depending on your application of course.</p>
<p>I will discuss how does docker and kubernetes at large, and how the containers orchestrated via them handle signals very briefly in this blog post.</p>
<h2 id="what-is-a-signal">What is a signal</h2>
<blockquote>
<p>A signal is a software interrupt and a way to communicate the state of a process(es) to another process, the OS and the hardware.</p>
</blockquote>
<p>By interrupt, we mean that whenever a signal is received by a process. It will stop doing whatever it is doing and handle it by either doing something about it or ignoring it.</p>
<p>A few Signals and what they intend to do(<a href="https://www.usna.edu/Users/cs/aviv/classes/ic221/s16/lec/19/lec.html">credits www.usna.edu</a>) are listed below.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Signal Value Action Comment
<span class="nt">----------------------------------------------------------------------</span>
SIGHUP 1 Term Hangup detected on controlling terminal
or death of controlling process
SIGINT 2 Term Interrupt from keyboard
SIGQUIT 3 Core Quit from keyboard
SIGILL 4 Core Illegal Instruction
SIGABRT 6 Core Abort signal from abort<span class="o">(</span>3<span class="o">)</span>
SIGFPE 8 Core Floating point exception
SIGKILL 9 Term Kill signal
SIGSEGV 11 Core Invalid memory reference
SIGPIPE 13 Term Broken pipe: write to pipe with no
readers
SIGALRM 14 Term Timer signal from alarm<span class="o">(</span>2<span class="o">)</span>
SIGTERM 15 Term Termination signal
SIGUSR1 30,10,16 Term User-defined signal 1
SIGUSR2 31,12,17 Term User-defined signal 2
SIGCHLD 20,17,18 Ign Child stopped or terminated
SIGCONT 19,18,25 Cont Continue <span class="k">if </span>stopped
SIGSTOP 17,19,23 Stop Stop process
SIGTSTP 18,20,24 Stop Stop typed at <span class="nb">tty
</span>SIGTTIN 21,21,26 Stop <span class="nb">tty </span>input <span class="k">for </span>background process
SIGTTOU 22,22,27 Stop <span class="nb">tty </span>output <span class="k">for </span>background process
</code></pre></div></div>
<p>Each signal has a default value and action defined. You can take a look at <code class="language-plaintext highlighter-rouge">sys/signal.h</code> and
find out more about each and every other signal defined in it.</p>
<h2 id="passing-the-signal-to-a-process-from-the-command-line">Passing the signal to a process from the command line</h2>
<p>When you do a <code class="language-plaintext highlighter-rouge">ctrl+c</code>, it’s the same as sending a <code class="language-plaintext highlighter-rouge">SIGINT</code> signal and when you type <code class="language-plaintext highlighter-rouge">ctrl+z</code>, it’s
the same as sending <code class="language-plaintext highlighter-rouge">SIGTSTP</code>, and when we type <code class="language-plaintext highlighter-rouge">fg</code> or <code class="language-plaintext highlighter-rouge">bg</code> that is the same as sending a
<code class="language-plaintext highlighter-rouge">SIGCONT</code> signal.</p>
<h2 id="passing-signals-to-containers">Passing signals to containers</h2>
<p>When you issue a <code class="language-plaintext highlighter-rouge">docker stop</code>, docker send <code class="language-plaintext highlighter-rouge">SIGTERM</code> to the process running as PID 1 inside the container
and waits for 10 seconds before it sends a <code class="language-plaintext highlighter-rouge">SIGKILL</code> to the kernel which will then straight terminate the
process, if the process hasn’t terminated within that time frame.</p>
<p>A <code class="language-plaintext highlighter-rouge">docker kill</code> will not give the container process, an opportunity to stop gracefully, but will straight ahead kill it.</p>
<h2 id="how-kubernetes-handles-signals">How kubernetes handles signals</h2>
<p>When you do a</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl delete pods mypod
</code></pre></div></div>
<p>it will send a <code class="language-plaintext highlighter-rouge">SIGTERM</code> and then wait for a set number of seconds to send a <code class="language-plaintext highlighter-rouge">SIGKILL</code> to the process,
this period is known as the grace termination period of the pod and can be configured in the <a href="https://kubernetes.io/docs/api-reference/v1.9/#podspec-v1-core">podSpec</a></p>
<p>If your process doesn’t handle <code class="language-plaintext highlighter-rouge">SIGTERM</code>, then it will get <code class="language-plaintext highlighter-rouge">SIGKILL</code>ed. Processes which are killed are
immediately removed from the etcd and the API, without waiting for the process to actually terminate on the
node.</p>
<p>If you have anything which needs a graceful shutdown, you need to implement a handler for <code class="language-plaintext highlighter-rouge">SIGTERM</code>.</p>
<p>If there are more than one containers in a pod, they both will be sent a <code class="language-plaintext highlighter-rouge">SIGTERM</code> and one would want
to implement the right strategy on how they should be terminated.</p>
<p>One mistake which is pretty common and something which can be missed is using the non-exec form of <a href="https://docs.docker.com/engine/reference/builder/#cmd"><code class="language-plaintext highlighter-rouge">CMD</code></a>
, for example, then your process is running as a child process of shell and not really running as root.
It will be running as <code class="language-plaintext highlighter-rouge">/bin/sh -c myapplication</code></p>
<p>The problem with this is that, shell will never forward this signal to the child process, which is your application
process and it will not be able to handle the <code class="language-plaintext highlighter-rouge">SIGTERM</code> for which you have written a handler for.</p>
<p>In any case, this would make your process be <code class="language-plaintext highlighter-rouge">SIGKILL</code>ed. Instead one should use the exec form CMD</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/bin/sh <span class="nt">-c</span> myapplication
</code></pre></div></div>
<p>or use the exec form of <a href="https://docs.docker.com/engine/reference/builder/#entrypoint"><code class="language-plaintext highlighter-rouge">ENTRYPOINT</code></a></p>
<h2 id="passing-custom-signals-to-container-processes">Passing custom signals to container processes</h2>
<p>Let’s take an example of a sample application, which has a dockerfile like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM alpine:3.5
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
RUN apk add --no-cache bash
ENTRYPOINT ["/entrypoint.sh"]
</code></pre></div></div>
<p>And the contents of your <code class="language-plaintext highlighter-rouge">entrypoint.sh</code> are</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nb">trap</span> <span class="s2">"echo TERM"</span> TERM
<span class="nb">trap</span> <span class="s2">"echo HUP"</span> HUP
<span class="nb">trap</span> <span class="s2">"echo INT"</span> INT
<span class="nb">trap</span> <span class="s2">"echo QUIT"</span> QUIT
<span class="nb">trap</span> <span class="s2">"echo USR1; sleep 2; exit 0"</span> USR1
<span class="nb">trap</span> <span class="s2">"echo USR2"</span> USR2
ps aux
<span class="nb">tail</span> <span class="nt">-f</span> /dev/null
</code></pre></div></div>
<p>Now if you build this container and run it in the background, if you try stopping the container
by doing <code class="language-plaintext highlighter-rouge">docker stop</code>. You will notice that the container process takes 10 seconds before the process
dies and get <code class="language-plaintext highlighter-rouge">SIGKILL</code>ed</p>
<p>To avoid that, we do a signal rewrite using <a href="https://github.com/Yelp/dumb-init">dumb-init</a> which runs a PID 1 for the docker
container. You can <a href="https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/">read here</a> about why running your application process as PID 1 is not usually a good idea.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM alpine:3.5
COPY entrypoint.sh /entrypoint.sh
RUN <span class="nb">chmod</span> +x /entrypoint.sh
RUN apk add <span class="nt">--no-cache</span> bash
<span class="c"># Change 1: Download dumb-init</span>
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN <span class="nb">chmod</span> +x /usr/local/bin/dumb-init
<span class="c"># Change 2: Make it the entrypoint. The arguments are optional</span>
ENTRYPOINT <span class="o">[</span><span class="s2">"/usr/local/bin/dumb-init"</span>,<span class="s2">"--rewrite"</span>,<span class="s2">"15:10"</span>,<span class="s2">"--"</span><span class="o">]</span>
CMD <span class="o">[</span><span class="s2">"/entrypoint.sh"</span><span class="o">]</span>
</code></pre></div></div>
<p>Now if you try issuing a <code class="language-plaintext highlighter-rouge">docker stop</code>, you will notice that the process exits out and it will get the
<code class="language-plaintext highlighter-rouge">USR1</code> signal and process it and then exit, which is what we wanted.</p>
<h2 id="signal-rewriting-for-gracefully-terminating-apache2-and-nginx">Signal rewriting for gracefully terminating apache2 and nginx</h2>
<p>If you want to initiate a graceful shutdown of an nginx server, you should send a <code class="language-plaintext highlighter-rouge">SIGQUIT</code>.
None of the Docker commands issue a <code class="language-plaintext highlighter-rouge">SIGQUIT</code> by default. So the solution for kubernetes is to
rewrite the signal <code class="language-plaintext highlighter-rouge">SIGTERM</code> to a <code class="language-plaintext highlighter-rouge">SIGQUIT</code> for nginx.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## for full source, check https://github.com/Yelp/casper/blob/master/Dockerfile.opensource</span>
FROM ubuntu:xenial
<span class="c"># Manually install dumb-init as it's not in the public APT repo</span>
RUN wget https://github.com/Yelp/dumb-init/releases/download/v1.2.1/dumb-init_1.2.1_amd64.deb
RUN dpkg <span class="nt">-i</span> dumb-init_<span class="k">*</span>.deb
<span class="c">## Your application requirements</span>
<span class="c"># Rewrite SIGTERM(15) to SIGQUIT(3) to let Nginx shut down gracefully</span>
CMD <span class="o">[</span><span class="s2">"dumb-init"</span>, <span class="s2">"--rewrite"</span>, <span class="s2">"15:3"</span>, <span class="s2">"/code/start.sh"</span><span class="o">]</span>
</code></pre></div></div>
<p>This will send the signal of <code class="language-plaintext highlighter-rouge">SIGQUIT</code> to nginx to gracefully handle termination</p>
<p>For gracefully terminating apache2, it requires a <code class="language-plaintext highlighter-rouge">SIGWINCH</code> signal to be passed to it. Which can be done by
passing the signal <code class="language-plaintext highlighter-rouge">28</code> to it.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## your dockerfile contents before this</span>
CMD <span class="o">[</span><span class="s2">"dumb-init"</span>, <span class="s2">"--rewrite"</span>, <span class="s2">"15:28"</span>, <span class="s2">"/code/start.sh"</span><span class="o">]</span>
</code></pre></div></div>
<h2 id="references">References</h2>
<ul>
<li><a href="https://lasr.cs.ucla.edu/vahab/resources/signals.html">https://lasr.cs.ucla.edu/vahab/resources/signals.html</a></li>
<li><a href="https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/">https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/</a></li>
<li><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods">https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods</a></li>
<li><a href="https://httpd.apache.org/docs/2.4/stopping.html">https://httpd.apache.org/docs/2.4/stopping.html</a></li>
<li><a href="https://www.usna.edu/Users/cs/aviv/classes/ic221/s16/lec/19/lec.html">https://www.usna.edu/Users/cs/aviv/classes/ic221/s16/lec/19/lec.html</a></li>
<li><a href="http://jkamenik.github.io/blog/2017/10/21/docker-details---dumb-init/">http://jkamenik.github.io/blog/2017/10/21/docker-details—dumb-init/</a></li>
<li><a href="http://nginx.org/en/docs/control.html">http://nginx.org/en/docs/control.html</a></li>
<li><a href="http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop">http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop</a></li>
<li><a href="https://github.com/Yelp/casper/issues/21">https://github.com/Yelp/casper/issues/21</a></li>
<li><a href="https://unix.stackexchange.com/questions/80044/how-signals-work-internally">https://unix.stackexchange.com/questions/80044/how-signals-work-internally</a></li>
<li><a href="https://stackoverflow.com/questions/37374310/how-critical-is-dumb-init-for-docker">https://stackoverflow.com/questions/37374310/how-critical-is-dumb-init-for-docker</a></li>
</ul>
Container Image Structuring for container runtimes
2019-04-10T00:00:00+00:00
https://www.tasdikrahman.com/2019/04/10/docker-rkt-containerd-container-image-structuring-for-container-runtimes
<p>While you might have read posts about <a href="https://news.ycombinator.com/item?id=16036268">docker</a> being dead, but given its adoption. That’s not <a href="https://sysdig.com/blog/2018-docker-usage-report/">really the case</a>.</p>
<p>While we have other container runtimes like <a href="https://github.com/opencontainers/runc">runc</a>,
<a href="https://blog.docker.com/2017/08/what-is-containerd-runtime/">containerd</a>,
<a href="https://coreos.com/rkt/">rkt</a> and some others. Docker is still something which a lot of
folks running containers use as their container runtime.</p>
<p>What this post will describe is one of the many approaches of structuring your container images, keeping in mind reusability, security and best practices in mind and keeping them as lightweight as possible. At the time of writing this, this is something which is still used to run production container workloads in my last company.</p>
<h3 id="prelude">Prelude</h3>
<p>Before going ahead, just so that we are on the same page.</p>
<blockquote>
<p>A Container image is a filesystem tree that includes all of the requirements for running a container,
as well as metadata describing the content. You can think of it as a packaging technology.</p>
</blockquote>
<blockquote>
<p>A container is composed of two things: a writable filesystem layer on top of a container image,
and a traditional linux process. Multiple containers can run on the same machine and share the OS
kernel with other containers, each running as an isolated processes in the user space. Containers
take up less space than VMs (application container images are typically tens of MBs in size), and
start almost instantly.</p>
</blockquote>
<p><a href="https://web.archive.org/web/20180128191607/http://docs.projectatomic.io/container-best-practices/">Source: project atomic, container best practices</a></p>
<h3 id="introduction">Introduction</h3>
<p><a href="http://martinfowler.com/bliki/ImmutableServer.html">Immutable Server</a> pattern is something which
we used to follow in my last company. <a href="https://medium.com/netflix-techblog/how-we-build-code-at-netflix-c5d9bd727f15">Netflix has written at length</a>
on how they do it. More on how we used to do it in another post.</p>
<p>I will not go into the how and why of immutable infra in this blog post, as that is something which
deserves its own post.</p>
<p>Docker presents fit’s right in if you follow the above pattern for your infrastructure.</p>
<p>Which is, if you are baking the whole application using <a href="https://www.packer.io/">packer</a> or something similar, including config inside the <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">AMI</a>,and then adding that AMI in the launch config for the <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html">ASG</a> so that the newer instance which comes up when the ASG scales up, is an exact copy of the instances already present in the ASG.</p>
<p>What you have is repeatable infra in short, with the above pattern. And you start treating servers as
<a href="https://news.ycombinator.com/item?id=7311704">cattle and not pets</a>.</p>
<h3 id="the-layering-of-container-images">The layering of container images</h3>
<center><img src="/content/images/2019/04/container-image-layering.png" /></center>
<p>We used to follow a layered approach of immutable infrastructure, where we would have a base layer.</p>
<h3 id="base-layer">Base Layer</h3>
<p>contains a fresh copy of an operating system (<a href="https://alpinelinux.org/">alpine Linux</a> in this case)
and would include core system tools, (eg: such as bash or coreutils, curl, <a href="https://github.com/Yelp/dumb-init">dumb-init</a> et al)
and tools necessary to install packages and make updates to the image over time.</p>
<p>As for the Intermediate container images, each would use the base layer, hence inheriting from the
base image.</p>
<h3 id="intermediate-layers">Intermediate Layers</h3>
<ul>
<li>Language runtime
<ul>
<li>python-27</li>
<li>php-7.1</li>
<li>go-1.8.3</li>
</ul>
</li>
<li>Web server
<ul>
<li>apache2</li>
<li>nginx</li>
</ul>
</li>
<li>Combination of (specific web server + specific language runtime)
<ul>
<li>python-27-{ nginx/apache2 }</li>
<li>php7-nginx-{ nginx/apache2 }</li>
<li>golang-nginx-{ nginx/apache2 }</li>
</ul>
</li>
</ul>
<p><em>Note: The above intermediate layers are just to show you an example, you can
replace it with your use case.</em></p>
<p>Dependency managers like pip/composer/golang-dep would go in this layer for the next layer
to make use of it and after their use we can clear their cache.</p>
<p>For example in the case of</p>
<ul>
<li>pip : <code class="language-plaintext highlighter-rouge">rm -rf ~/.cache/pip/*</code></li>
<li>composer : <code class="language-plaintext highlighter-rouge">composer clear-cache</code></li>
<li>apk: <code class="language-plaintext highlighter-rouge">rm -rf /var/cache/apk/*</code> that is if <code class="language-plaintext highlighter-rouge">--no-cache</code> is not being passed whilst <code class="language-plaintext highlighter-rouge">apk add <package></code></li>
<li>go-dep: the cache might be useful for debugging if something went wrong with that
old cache. But this can be debated of whether or not to remove <code class="language-plaintext highlighter-rouge">$GOPATH/pkg/dep</code></li>
</ul>
<h3 id="an-example-of-such-a-setup">An example of such a setup</h3>
<ul>
<li>Base layer</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM gliderlabs/alpine:3.4
ENV ALPINE_VER=3.4
ENV ALPINE_SHA=45ba65c1116aaf668f7ab5f2b3ae2ef4b00738be
RUN apk update && \
apk add xorriso git xz curl tar iptables cpio bash && \
rm -rf /var/cache/apk/*
RUN apk add -U --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing aufs-util
RUN addgroup -g 2999 docker
</code></pre></div></div>
<p>after which you would create the container image from this Dockerfile. And for the sake of this example, you would name it as <code class="language-plaintext highlighter-rouge">base-image</code></p>
<ul>
<li>Intermediate Layer</li>
</ul>
<p>To create a JAVA based intermediate layer</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM tasdikrahman/base-layer:0.1.0
ENV LANG=C.UTF-8
RUN curl -LO 'http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jre-8u131-linux-x64.tar.gz' -H 'Cookie: oraclelicense=accept-securebackup-cookie' \
&& chown root:root jre-8u131-linux-x64.tar.gz \
&& tar -xzf jre-8u131-linux-x64.tar.gz \
&& rm jre-8u131-linux-x64.tar.gz \
&& mv jre1.8.0_131 /usr/local/lib
WORKDIR /usr/local/lib/jre1.8.0_131
ENV JAVA_HOME /usr/local/lib/jre1.8.0_131
ENV PATH $JAVA_HOME/bin:$PATH
RUN apk del --no-cache curl tar # wget ca-certificates
</code></pre></div></div>
<ul>
<li>Application Layer</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM tasdikrahman/java-base:0.1.0
# Your application specific requirements etc.
</code></pre></div></div>
<h3 id="application-image-layer">Application Image layer</h3>
<p>This is where the container image would contain dependencies specific to the application and other
required tooling, inheriting other things from the previous layers.</p>
<h3 id="security">Security</h3>
<ul>
<li>
<p><a href="https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.html">Dumb-init</a>
should be specified as the entrypoint for the application container image which is yet
to be followed in some remaining container images so that /entrypoint.sh is executed as
CMD as an argument to dumb-init. <a href="https://news.ycombinator.com/item?id=11802993">More on why have something like dumb-init as PID 1</a></p>
</li>
<li>
<p>If the service does not need <code class="language-plaintext highlighter-rouge">root</code> privileges, create a new user and switch the user with <code class="language-plaintext highlighter-rouge">USER</code>
directive in the application container image.</p>
</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RUN groupadd -r myapp && useradd -r -g myapp myapp
USER myapp
</code></pre></div></div>
<ul>
<li>Adding better security vulnerability testing
<ul>
<li><a href="https://github.com/jmccann/drone-clair">Drone Clair Plugin</a></li>
</ul>
</li>
</ul>
<h3 id="keeping-the-size-of-the-docker-image-small">Keeping the size of the docker image small</h3>
<p>At each layer</p>
<ul>
<li>the necessary package managers should be cleared of their cache</li>
<li>Unnecessary layers file system layers should not be created</li>
<li>Unwanted packages/libs should not be added.</li>
</ul>
<p>The above division ideally, should always be maintained and any new requirement should
always go into the layer most appropriate for it</p>
<ul>
<li>
<p>Use <code class="language-plaintext highlighter-rouge">.dockerignore</code> wherever necessary as when building the image, docker has to prepare
context first, gather all files which would be used in a process. Default context contains
all files in the directory, which would include things like .git directory for example. Which
can get pretty big citing the .git/objects subdir.</p>
</li>
<li>Optimizing <code class="language-plaintext highlighter-rouge">COPY</code> and <code class="language-plaintext highlighter-rouge">RUN</code> directives by putting least frequently changed things on the top of the Dockerfile, which would help us enable caching better.</li>
<li>Whenever possible, chaining commands together (if possible) and sorting multi-line arguments alphanumerically, which will help avoid duplication of packages and make the list much easier to update. This also makes it a lot easier to read and review. Adding a space before a backslash () helps as well.</li>
</ul>
<p>Example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RUN apt-get update && apt-get install -y \
bzr \
cvs \
git \
mercurial \
subversion
</code></pre></div></div>
<h3 id="good-to-have">Good to have</h3>
<ul>
<li>Linting, which can enforce standardization across the container images, a possible solution will be https://github.com/projectatomic/dockerfile_lint</li>
</ul>
<h3 id="references">References</h3>
<ul>
<li>http://docs.projectatomic.io/container-best-practices/</li>
<li>https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/</li>
<li>https://opensource.googleblog.com/2018/01/container-structure-tests-unit-tests.html</li>
<li>https://github.com/Yelp/dumb-init</li>
</ul>
Self hosting kubernetes
2019-04-04T00:00:00+00:00
https://www.tasdikrahman.com/2019/04/04/self-hosting-highly-available-kubernetes-cluster-aws-google-cloud-azure
<p><a href="https://kubernetes.io/">kubernetes</a> has been around for some time now. At the time of writing this article, <a href="https://github.com/kubernetes/kubernetes/releases/tag/v1.14.0">v1.14.0</a> is the latest release and with each new release, they have a bunch of new features.</p>
<p>This post is about the initial setup for getting the kubernetes cluster up and running and assumes that you are already familiar with what kubernetes is and a rough idea on what the control plane components are and what do they do. I gave a talk on the same subject of self-hosting kubernetes in <a href="https://devopsdaysindia.org/">DevOpsDays India, 2018</a></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/3WgqFoo9eek" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>You can find the slides of the talk above.</p>
<script async="" class="speakerdeck-embed" data-id="40484a078640415a872c2857fd7aaf89" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>
<p>Although there are other tools like <a href="https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/">kubeadm</a> which would help you to <a href="https://thenewstack.io/kubernetes-now-does-self-hosting-with-kubeadm/">self-host</a> kubernetes, I would like to show how <a href="https://github.com/kubernetes-incubator/bootkube">bootkube</a> does it.</p>
<h2 id="what-is-self-hosting-kubernetes">What is self-hosting kubernetes?</h2>
<blockquote>
<p>It runs all required and optional components of a Kubernetes cluster on top of Kubernetes itself. The kubelet manages itself or is managed by the system init and all the Kubernetes components can be managed by using Kubernetes APIs.</p>
</blockquote>
<p><em>source: <a href="https://coreos.com/tectonic/docs/latest/troubleshooting/bootkube_recovery_tool.html">CoreOS tectonic docs</a></em></p>
<p>In a nutshell, static Kubernetes runs control plane components as <a href="https://www.freedesktop.org/wiki/Software/systemd/">systemd</a> services on the host. Its simple to reason about and the repo has educational docs, but in practice, it’s fairly static (hard to re-configure). Self-hosted Kubernetes runs control plane components as pods. A one-time bootstrapping process is done to set up that control plane. Configuring hosts becomes much more minimal, requiring only a running Kubelet. This favours performing rolling upgrades through Kubernetes, the cluster system, and provisioning immutable host infrastructure. A node’s only job is to be a “dumb” member of the larger cluster.</p>
<center><img src="/content/images/2019/04/k8s-self-hosted.png" /></center>
<p>The cluster which you see above is a self-hosted cluster, hosted on <a href="https://digitalocean.com">digitalocean</a> using <a href="https://github.com/poseidon/typhoon">typhoon</a>. For reference, you can check the <a href="https://github.com/tasdikrahman/infra/tree/master/aws/ap-south-1/homelab">terraform</a> config to bring up your own cluster using typhoon.</p>
<p>What you can see above is how you can use <a href="https://kubernetes.io/docs/reference/kubectl/overview/">kubectl</a> to interact with the different components of the kubernetes cluster and how they are abstracted in terms of the native kubernetes objects like <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">deployments</a>, <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">daemonset</a></p>
<h2 id="is-this-something-new">Is this something new?</h2>
<center><img src="/content/images/2019/04/k8s-self-hosted-github-proposal.png" /></center>
<p>This has been in discussion for quite some time and a lot of people have already done it in <a href="https://coreos.com/tectonic/">production</a></p>
<h2 id="why">Why?</h2>
<p>What are the usual properties of the <a href="https://kubernetes.io/docs/concepts/#kubernetes-control-plane">k8s control plane</a> objects.</p>
<ul>
<li>Highly available</li>
<li>Should be able to tolerate node failures</li>
<li>Scale up and down with requirements</li>
<li>Rollback and upgrades</li>
<li>Monitoring and alerting</li>
<li>Resource allocation</li>
<li>RBAC</li>
</ul>
<h2 id="how-is-self-hosted-kubernetes-addressing-them">How is self-hosted kubernetes addressing them</h2>
<ul>
<li><strong>Small Dependencies</strong>: self-hosted should reduce the number of components required, on the host, for a Kubernetes cluster to be deployed to a Kubelet. This should greatly simplify the perceived complexity of Kubernetes installation.</li>
<li><strong>Deployment consistency</strong>: self-hosted reduces the number of files that are written to disk or managed via configuration management or manual installation via SSH. Our hope is to reduce the number of moving parts relying on the host OS to make deployments consistent in all environments.</li>
<li><strong>Introspection</strong>: internal components can be debugged and inspected by users using existing Kubernetes APIs like kubectl logs</li>
<li><strong>Cluster Upgrades</strong>: Related to introspection the components of a Kubernetes cluster are now subject to control via Kubernetes APIs. Upgrades of Kubelet’s are possible via new daemon sets, API servers can be upgraded using daemon sets and potentially deployments in the future, and flags of add-ons can be changed by updating deployments, etc</li>
<li><strong>Easier Highly-Available Configurations</strong>: Using Kubernetes APIs will make it easier to scale up and monitor an HA environment without complex external tooling. Because of the complexity of these configurations tools that create them without self-hosted often implement significant complex logic.</li>
<li><strong>Streamlined, cluster lifecycle management</strong>: you can manage things using your favourite tool like kubectl</li>
</ul>
<p>Let’s try explaining the above one at a time.</p>
<h4 id="small-dependencies">Small dependencies</h4>
<center><img src="/content/images/2019/04/small-deps.png" /></center>
<ul>
<li>Forget about masters for a second, for worker nodes, the components above is all you need for it to connect to the cluster.</li>
<li>Everything is running inside kubernetes.</li>
<li>Master nodes might have systemd units and some other specialised scripts to run.</li>
<li>There is no distinction between the nodes.</li>
</ul>
<center><img src="/content/images/2019/04/k8s-disctinction.png" /></center>
<p>The nodes are only differentiated on the basis of k8s <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/">labels</a> which are attached to nodes and are the only way one can distinguish between the master the other kinds of nodes.</p>
<p>This is done in order so that the scheduler can schedule only the required workloads on the nodes of a particular kind, for example. The API server should only be running on the master nodes, and hence it will try using the label which is attached to the master nodes, tolerate the taint and get scheduled on the master nodes.</p>
<p>Adding labels to a node is as simple as doing a</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kubectl label node node1 master=true
</code></pre></div></div>
<h4 id="introspection">Introspection</h4>
<center><img src="/content/images/2019/04/introspection.png" /></center>
<p>All the control plane objects run as one or the other kubernetes objects, what happens then is you get the power of doing something like <code class="language-plaintext highlighter-rouge">kubectl logs ..</code> on the particular object to get the logs.</p>
<p>You can go ahead and send these logs to your logging platform just like how you would do to your application logs. Making them searchable and gaining more visibility on the control plane objects.</p>
<h4 id="cluster-upgrades">Cluster Upgrades</h4>
<center><img src="/content/images/2019/04/upgrades.png" /></center>
<p>Doing a cluster upgrade for a particular component is as simple as doing a <code class="language-plaintext highlighter-rouge">kubectl set-image</code> on the particular control plane object.</p>
<p>First comes the API server. A <a href="https://github.com/kubernetes-incubator/bootkube/blob/master/Documentation/upgrading.md">certain flow</a> needs to be replicated while doing upgrades, but other than that. This is really just it for upgrades</p>
<h4 id="easier-highly-available-configurations">Easier highly available configurations</h4>
<center><img src="/content/images/2019/04/easier-ha.png" /></center>
<p>As the core control plane objects are running as kubernetes objects like deployments, you can increase their replica count and make it scale up/down to the desired number which you would want.</p>
<p>Wouldn’t really affect the scheduler/controller-manager/api operations too much if you are thinking of scaling them up and expecting performance improvement as each of them take a lock on etcd and elect a leader, hence only one of them is active at any given point.</p>
<h4 id="streamlined-cluster-lifecycle-management">Streamlined cluster lifecycle management</h4>
<p>Like any of your apps, your control plane objects can be applied to your cluster in a similar way.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kubectl apply -f kube-apiserver.yaml
$ kubectl apply -f controller-manager.yaml
$ kubectl apply -f flannel.yaml
$ kubectl apply -f my-app.yaml
</code></pre></div></div>
<p>Which gives you a familiar ground to be in, of course, you should try not doing the above in prod and have some kind of automation to do things instead of doing <code class="language-plaintext highlighter-rouge">kubectl apply</code>.</p>
<h2 id="how-does-all-this-work">How does all this work?</h2>
<p>There are three main problems to solve here</p>
<center><img src="/content/images/2019/04/main-problems.png" /></center>
<h4 id="bootstrapping">Bootstrapping</h4>
<ul>
<li>Control plane running as daemonsets, deployments. Making use of secrets and configmaps</li>
<li>But … We need a control to plane to apply these deployments and daemonsets on</li>
</ul>
<center><img src="/content/images/2019/04/chicken.png" /></center>
<h2 id="how-to-solve-this">How to solve this?</h2>
<p><a href="https://github.com/kubernetes-incubator/bootkube">Bootkube</a> can be used to create the temporary control plane which can be then used to inject the control plane objects.</p>
<h2 id="how-does-bootkube-work">How does bootkube work?</h2>
<p>A very rough way of describing the set of operations</p>
<ul>
<li>Bootkube starts, and drops api-server, scheduler, controller-manager, and etcd pod manifests into: <code class="language-plaintext highlighter-rouge">/etc/kubernetes/</code> manifests.</li>
<li>This static (and temporary) control-plane starts, then bootkube injects daemonsets, deployments, secrets, & configMaps which make up the self-hosted control-plane.</li>
<li>Bootkube waits for the self-hosted control-plane to start, and when all required pods are running, bootkube deletes the static manifests in <code class="language-plaintext highlighter-rouge">/etc/kubernetes/manifests</code> and exits (leaving us with a self-hosted cluster)</li>
</ul>
<center><img src="/content/images/2019/04/bootkube-01.png" /></center>
<center><img src="/content/images/2019/04/bootkube-02.png" /></center>
<center><img src="/content/images/2019/04/bootkube-03.png" /></center>
<p>So you have manifests in two directories</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">bootstrap-manifests</code></li>
</ul>
<p>which would hold the temporary control plane manifests, which bootkube would bring up</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">manifests</code></li>
</ul>
<p>which would be the manifests applied, when the API server pivots from the temporary one to the one which was injected and applied when bootkube exited.</p>
<center><img src="/content/images/2019/04/manifests.png" /></center>
<h3 id="does-this-even-work">Does this even work?</h3>
<center><img src="/content/images/2019/04/controller-manager.png" /></center>
<p>What you see above are the logs of a controller node when the cluster is brought up. The initial <code class="language-plaintext highlighter-rouge">docker ps -a</code> would show the bootstrap control plane objects which were brought up bootkube.</p>
<p>The second <code class="language-plaintext highlighter-rouge">docker ps -a</code> shows the list of containers which are part of the pods when all the other manifests in the <code class="language-plaintext highlighter-rouge">manifests</code> dir got applied.</p>
<h3 id="tools-to-self-host-k8s-clusters">Tools to self host k8s clusters</h3>
<p>Although I have used typhoon to bring up my kubernetes clusters, check out <a href="https://gardener.cloud/">gardener cloud</a> which can also help you achieve self hosted kubernetes.</p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/self-hosted-kubernetes.md">SIG-lifecycle Spec on self hosted kubernetes</a></li>
<li><a href="https://github.com/kubernetes-incubator/bootkube/blob/master/Documentation/design-principles.md">bootkube: Design principles</a></li>
<li><a href="https://blog.rmenn.in/post/how-bootkube-works/">bootkube: How does is work</a></li>
<li><a href="https://github.com/kubernetes-incubator/bootkube/blob/master/Documentation/upgrading.md">bootkube: Upgrading the kubernetes cluster</a></li>
<li><a href="https://groups.google.com/forum/#!msg/kubernetes-sig-cluster-lifecycle/p9QFxw-7NKE/jeYJF1hBAwAJ">SIG lifecycle google groups early discussions on self hosting</a></li>
<li><a href="https://github.com/poseidon/typhoon">Typhoon</a></li>
<li><a href="https://www.youtube.com/watch?v=jIZ8NaR7msI">Self hosting kubernetes: how and why</a></li>
</ul>
Solo Backpacking trip to Hampi, Gokarna and Goa
2019-03-22T00:00:00+00:00
https://www.tasdikrahman.com/2019/03/22/solo-backpacking-trip-to-hampi-gokarna-goa-budget
<p>It was a cold Friday night, had just come back home after wrapping up the farewell party which was thrown by my colleagues from my last company. I was feeling worn out of all the activities from the day, but I kept reminding myself that I had to start packing for the impromptu solo backpacking trip which I had planned before I joined my next gig in the coming 10 days.</p>
<center><img src="/content/images/2019/03/packing.jpg" /></center>
<h3 id="where-was-i-planning-to-go">Where was I planning to go?</h3>
<iframe src="https://uploads.knightlab.com/storymapjs/f8903d09026b4be3c6e94eca06c8e73e/hampi-gokarna-goa/index.html" frameborder="0" width="100%" height="800"></iframe>
<h3 id="hampi-the-imperial-capital-of-vijayanagar-a-14th-century-empire">Hampi the imperial capital of Vijayanagar, a 14th century empire</h3>
<p>Hampi is the forgotten empire, there was a time when diamonds were sold on the streets before it’s fall. Such was the grandeur.</p>
<p>Hampi rose into prominence in the early 14th century when the Kampili Kings rose to power. In 1327, the kingdom was attacked by Muhammad-bin-Tughluq who took two brothers, Bukka and Harihara as prisoners along with thousands of other people. These brothers tricked the Sultan into setting them free and returned to Kampili to set up a kingdom of their own with its capital at Vijayanagara. Thus the Vijayanagara Empire was founded by Harihara I and Bukka I of the Sungama dynasty in 1336. The Sungama dynasty was followed by the Saluvas and the Tuluvas each of whom added Vijayanagara’s architectural beauty.</p>
<p>At one point Hampi was also one of the biggest trading centres of the world. Vijayanagar brought a lot of wealth, fame and splendour to Hampi. In those times, most markets in Hampi were always crowded and swarming with buyers and also merchants.</p>
<p>The empire encompassed almost the whole of modern South India.</p>
<p>It is quite a unique experience I would say. On one hand, you experience a culture rich in history(every street and corner is steeped in history) and on the other hand on crossing the river, you enter into another world — Virupapur Gadde or more popularly known as the Hippie Island. Contradictory indeed! You will find the street lined with cafes which cater to foreign tourists from all over the world. Tourists from Israel are a common sight.</p>
<p>The easiest way to reach Hampi is to take a bus till Hospet. After which you can take a city bus to the town of Hampi. Which barely costs you anything. If you are feeling lucky, you can book a Karnataka sarige class, KSRCTC bus from Bangalore’s Majestic bus stand, till Hospet like I did. But a fair warning is that, if you have a big rucksack with you and are looking for a good night’s sleep(which I feel everyone does), you are better off booking a better bus service for yourself. But nonetheless, it’s a very good service, provided it charged me just about 230 INR, for a distance of close to 300 km.</p>
<center><img src="/content/images/2019/03/hampi-first-impressions.jpg" /></center>
<p>For staying around Hampi, you can book one of the lodges which are nothing but the cafe’s around the <a href="https://en.wikipedia.org/wiki/Virupaksha_Temple,_Hampi">Virupaksha temple</a>, the prices are quite cheap. I booked myself a room for a meagre 450INR for two nights in of the nearby lodges. The one which I was staying in, had bathrooms which were not well maintained at all. I was lucky that there was a public bathroom near the lodge, where I used to get done with my business. But given the price of the lodge, I wouldn’t have expected any better.</p>
<p>There is one particular food joint just right next to the lodge, which used to serve <a href="https://en.wikipedia.org/wiki/Dosa">Dosas</a> and I used to get breakfast and dinner, along with some tea. The smell of the batter getting cooked on the stove and it getting served on your plate with some hot sambar. Heavenly!</p>
<p>The plan for the very first day was to get a cycle (they can be rented at around 100/200 INR for the whole day) and cover the area near Virupaksha temple and the road which leads to the southern parts of the ancient city.</p>
<center><img src="/content/images/2019/03/hampi-virupaksha.jpg" /></center>
<p>It’s part of the group of monuments at Hampi, designated as a UNESCO world heritage site.</p>
<p>The day on which I visited, it just so happened that there was a marriage ceremony going on. There was a huge crowd which is as expected. Lakshmi, the temple elephant would be seen blessing the visitors with her trunk. I would later get to know that it’s quite common for weddings to happen here.</p>
<p>Right next to the temple, is the Hemakuta hill which leads to other temples in the vicinity. The hill encircled on its three sides by massive fortification. To the north is the enclosure wall of Virupaksha temple. The complex has three gateways. More than 30 shrines stand on this hill. These vary from elaborate structures with multiple sanctums and rudimetaries, single-celled construction. Most of these temples have stepped pyramidal type of superstructure.</p>
<center><img src="/content/images/2019/03/hemakuta-hill.jpg" /></center>
<center><img src="/content/images/2019/03/sunset-point.jpg" /></center>
<center><img src="/content/images/2019/03/krishna-temple.jpg" /></center>
<p>There is a part of the empire, which used to be the palace grounds of the king. It had a part where the sacred water tanks are situated. The sacred tanks were related to various rituals and functional aspects of the temples and the people surrounding the temples. The tanks were considered to be sacred places by the people of Hampi in ancient times.</p>
<p>There are some water tanks that are not related to the temples. Some of the water tanks are situated within the Royal Enclosure and they were built for the use of the members of the royal family of Vijayanagara. There were a few large public water tanks as well that were for use of the general people.</p>
<p>The pushkaranis in Hampi was an integral part of the people’s lives during the time of the Vijayanagara Empire. Since the temples were an important part of the social and cultural lives of the people of ancient Hampi, the water tanks also gained significance among the people.</p>
<p>In many cases, the water tanks served as the venue for the annual boat festivals. During such festivals, the images of Gods and Goddesses were taken out of the temples for a coracle ride on the water tanks. Sometimes the images were placed on the pavilions that can be seen in the middle of some of the pushkaranis.</p>
<center><img src="/content/images/2019/03/pushkarani.jpg" /></center>
<p>The one you see in the pictures, is the pushkarani of the royal enclosure. The key attraction of the tank is the symmetrical layout of the steps. It is a 5 tiered tank where each tier comprises of a few steps.</p>
<center><img src="/content/images/2019/03/drainage.jpg" /></center>
<p>The soldiers used to eat their meals in these giant stone utensils, that’s about 5 times the size of an average eating utensil which you can find at a regular household these days. The city had it’s drainage system even back then, which is something very rare in civilisations of that time.</p>
<center><img src="/content/images/2019/03/hampi-cycling.jpg" /></center>
<p>The structure which you see on the top is called the Mahanavmi Dibba, a three tired stone platform, rising to a height of 8 meters. and is located to the north-east of the royal enclosure. It was one of the most important ceremonial structures of royal used, built in granite and subsequently encased in sculptured schist stone. It is dated circa 16th century AD.</p>
<p>There are references to the use of the platform by the royal family, for important festivals like Mahanavami, by Abdur Razak and Domingo Paes.</p>
<p>Right behind the Virupaksha temple, is the massive Tungabhadra river flowing. It’s quite a sight during sunset. The common folks of the village cross the river to the other side of the river using the circular boats which look really cute. More on that in a little bit.</p>
<center><img src="/content/images/2019/03/virupaksha-backyard.jpg" /></center>
<center><img src="/content/images/2019/03/thungabhadra-river.jpg" /></center>
<p>One of the best parts of travelling is that I get to have conversations with folks whom I generally wouldn’t have gotten to have a chat with. This time being no different.</p>
<p>I met Yang, who hails from Japan. When I met him, he already had been travelling for quite some time. He had been hiking across India for quite some time now and his next step is going to Kochi. He had quit his job in an investment bank in Tokyo last year and had been travelling ever since.</p>
<center><img src="/content/images/2019/03/yang.jpg" /></center>
<p>The Virupaksha temple, as it looks like during sunset.</p>
<center><img src="/content/images/2019/03/virupaksha-temple.jpg" /></center>
<h2 id="day-2---hampi">Day 2 - Hampi</h2>
<p>The second day came pretty quickly, usual things. Getting breakfast at the same place. All prepped for the trek towards the eastern side of Hampi.</p>
<center><img src="/content/images/2019/03/day-2-hampi.jpg" /></center>
<p>The hike was a 3 kilometre trek from my lodge, the Vitthala group of temples.</p>
<center><img src="/content/images/2019/03/day-2-vitthala.jpg" /></center>
<p>The path to the temple is not that hard IMO if you are alright with walking in the sun with little to zero presence of humans in the path under the scorching sun. You walk alongside the river banks in some parts and get to witness the ruins which were left behind when the empire had fallen.</p>
<p>The other way of reaching to the Vitthala temple from Virupaksha is to take a 12 km auto ride which would push you back by some 800 INR or so depending on how well you can bargain with the auto drivers.</p>
<p>Whichever suits you well. I wanted to take that hike as my lodge owner had mentioned it should be doable given I had lots of water.</p>
<center><img src="/content/images/2019/03/stone-chariot.jpg" /></center>
<p>The highlight of the temple is the stone chariot, at the entrance, a reproduction of a professional wooden chariot, is perhaps the most stunning achievement, typical of the Vijayanagar period. It houses an image of Garuda, the vehicle of Lord Vishnu.</p>
<center><img src="/content/images/2019/03/vitthala-temple-chariot.jpg" /></center>
<p>Outside the temple, to the east, is a huge bazaar, measuring 945 meters in length and 40 meters in width leading to a sacred tank known as lokapavani.</p>
<center><img src="/content/images/2019/03/hampi-market.jpg" /></center>
<p>If you keep walking in that direction, you will what is called <a href="http://hampi.in/talarigatta-gate">Talarigatta gate</a>, which is located strategically to the north, north-east of the Vijayanagara city leading to the river known as ‘Talarigatta’(toll collection point). The gateway, which is narrow, is built into the fortification wall which enclosed the capital city. It’s a two-storied structure with a provision for a guard pavilion in the first storey, the latter having beautifully cut plaster decoration.</p>
<center><img src="/content/images/2019/03/talarigatta.jpg" /></center>
<p>Just after that, I hitchhiked with Pasha to the nearest main road to Kamalapur, which is the nearest town coming in between Hampi and Nimbapura. Was around 4kms. Pasha has a soft drink ship right outside the Vitthala temple and was going out to deposit the empty glasses to the distributor when I met him. He has two kids, named Aminah and Aliyah whom he adored. He asked me about my family during the ride and was describing how he loved Bombay and the Haji Ali Dargah. He complained about the traffic there though.</p>
<p>Here’s a picture of Pasha and me, which I managed to get with him just after he dropped me.</p>
<center><img src="/content/images/2019/03/pasha.jpg" /></center>
<p>I got dropped very near to the museum, so the natural thing was to just go and visit it. It was interesting to see a lot of sculptures and old coins from the empire. And for a change, I was not out in the scorching sun.</p>
<p>As I was heading back to the main Hampi city, I had to find my way back. I had a few options. The easiest was to take an auto and head back to the city. Or wait for the red city bus to come along and halt there at that intersection. I waited for the bus to come and it felt like eons, but it didn’t arrive.</p>
<p>While I was waiting there, I met this kid called Mahadev whom I had met just yesterday when I had visited Virupaksha. He was one of the many small kids who was selling the map of Hampi in front of the temple. And I could immediately recognize him. He was waiting for the same bus as I was and he was on his way back home after a regular day at school.</p>
<center><img src="/content/images/2019/03/mahadev.jpg" /></center>
<p>And while we waited, we talked about his family. His father was an auto driver here and his mom used to sell food at her stall near the Virupaksha temple grounds. He had an elder sister who was studying in the next town. I asked him what does he like about this place. He said. it was the people and that he liked meeting new people and getting to know them.</p>
<p>After waiting for a long while and both of us sharing some apples which I was carrying, Mahadev decided he would get on the back of a motorcycle and reach the town, he generously offered me his ride when a kind gentlemen offered to give him a lift, but I asked him to take the ride as I was hoping to get something similar.</p>
<p>As luck would have it, generous gentlemen offered me a ride back into the city. He struck up a conversation immediately after we started towards the city, asking me about where I was from and how was I liking the city.</p>
<p>The next thing on the list was to get on the circular boats which I saw when I hiking down the river. The owner of the boating service was quoting somewhere around 1200 INR for around an hour. I managed to bring it down to some 400INR after a lot of cajoling.</p>
<center><img src="/content/images/2019/03/circular-boat.jpg" /></center>
<p>Later that night, I was lazing around the same local tea shop where I used to hang around. I ended up eating at the same place. Took a bus back to Hospet after which I had to get on a bus from Hospet on to my next destination.</p>
<h3 id="another-uncomfortable-bus-ride">Another uncomfortable bus ride</h3>
<p>The bus ride to <a href="https://en.wikipedia.org/wiki/Gokarna,_Karnataka">Gokarna</a> was not exactly the best ride, but it was pretty cheap. I was travelling in the state bus again, no surprises here that I couldn’t get any sleep again.</p>
<p>I met Yang, yet again while on my way to Gokarna. And we had a chat again about where we were headed and what are we gonna do next.</p>
<center><img src="/content/images/2019/03/yang-day2.jpg" /></center>
<p>The trip to Gokarna from Hampi is an overnight one and when I woke up to the morning, I was pleasantly surprised by the gentle breeze on my face flowing in through the tiny crevice of the window which slid open as the bus crawled through the bridge.</p>
<p>I got down at the Gokarna city bus stand and decided to walk all the way up to the hostel which I had booked for the stay for one night. It was a kilometre of a hike before I could reach my hostel.</p>
<p>I was welcomed by the hosts back at the hostel and the hostel was right next to the ridge where you could have a clear view of the <a href="https://www.tripadvisor.com.sg/LocationPhotoDirectLink-g651646-d2651456-i55075183-Gokarna_Beach-Gokarna_Dakshina_Kannada_District_Karnataka.html">main beach</a> and the view was absolutely gorgeous!</p>
<center><img src="/content/images/2019/03/gokarna-zostel.jpg" /></center>
<p>The bunk bed which I was sharing along, had folks who were there for a quick getaway before they attended a friend’s wedding, an army officer who was on her way back home before she got to go back to Leh where she was commissioned, I met Amazon, who was from <a href="https://en.wikipedia.org/wiki/Sussex">Sussex</a> and who was on a break before he headed back home and joined his new gig, along with Michael who worked as a Police officer in Australia.</p>
<center><img src="/content/images/2019/03/gokarna-day-1.jpg" /></center>
<p>I lazed around the dining area of the hostel which was right next to the ridge, where I and a couple of folks were just sharing with each other where they had been travelling around and singing songs while someone played some guitar sipping over some milkshake.</p>
<p>The afternoon was sliding over and we started to plan around a hike around the beaches of Gokarna.</p>
<p>We made it down from the hills from where the hostel was situated and headed to the main beach, where we boarded onto the speed boat which was waiting for us to take us to God’s own beach.</p>
<center><img src="/content/images/2019/03/gokarna-boat.jpg" /></center>
<p>The boat ride was roughly a 30-minute ride till we reached the destination and we were greeted with occasional dolphins who would swim right next to us and come out of the water every now and then.</p>
<center><img src="/content/images/2019/03/gods-own-beach.jpg" /></center>
<p>Just after you get down at any of the beaches which are farther away from the main roads, you would find a bunch of folks who would have holed up in makeshift tents, playing soccer, cooking food in the open kitchen which they have created. Most of them being from Israel.</p>
<p>We started our trek to the beach right next to God’s own beach and the beaches along the way, all the way to the main Gokarna beach.</p>
<p>Kudle Beach is popular with those that are staying in town but want to spend the day at the beach. At the left end of the Gokarna beach, a narrow path goes up a hill, where you cross a (Rama) temple en route. This temple also has a natural water spring which according to the locals never stops running. The water is quite drinkable. After climbing up some stairs, you will find flat ground and some breath-taking views of Gokarna beach as you turn around to see the distance you covered. As you move along, about 10 minutes walk from this place, the flat ground leads to a narrow lane, which goes down to Kudle beach, the second of Gokarna’s beaches. This beach looks very unkempt, desolate and dirty in off-seasons. You will hardly find a soul here then.</p>
<p>One of the beaches which we crossed was called the Om beach, it gets its name from the fact that, if you standing at the ridge. Just before Om beach. You can actually see the shape of <a href="https://en.wikipedia.org/wiki/Om">Om</a> being formed by the beach. On the way, we crossed the famous Namaste cafe and some others which had cropped up.</p>
<p>At the end of the Om beach, there is a path going up the hill. Here one has to get around a hillock(about 20-minutes walk) to reach Half-Moon beach. take this trail, and when you reach a fork in the trail, take a right for the coast route, and left for the forest route. They will both take you to the same place. Half-moon beach is so named because the shape resembles that of a half-moon.</p>
<p>At the end of the Half-moon Beach, a small trail leads to Paradise beach, also known as Full-moon Beach. It’s around 20 minutes walk from Half-moon Beach. The thing to remember here is after crossing the first set of rocks, one should not try to climb the hill. Rather try getting around the hill. Its a much easier climb. The steep climb up the hill will take you to the next village, Bellekan. This is the last of the Gokarna beaches.</p>
<center><img src="/content/images/2019/03/om-beach.jpg" /></center>
<center><img src="/content/images/2019/03/om-beach-2.jpg" /></center>
<p>Gokarna is a place where a lot of foreigners come during their season time. By season time, they mean when it’s too cold out there back in their home countries. I met this couple from Russia and how they used to come here every winter.</p>
<center><img src="/content/images/2019/03/om-beach-3.jpg" /></center>
<center><img src="/content/images/2019/03/om-beach-4.jpg" /></center>
<center><img src="/content/images/2019/03/om-beach-5.jpg" /></center>
<p>By the time we reached, the hilltop of the main Gokarna beach, it was already time for sunset. And I swear, it has been one of the most alluring sunsets I have witnessed in my life so far!</p>
<center><img src="/content/images/2019/03/gokarna-yoga.jpg" /></center>
<p>We could see, the sun getting gobbled up by the sea. The orange it was, shining bright. Spreading joy and heat, there were a few folks who were meditating, someone was drawing something and busy scribbling something on their scratch pad and here I was looking at the distant sea blankly. Just doing nothing but staring at it. I still remember that moment as I write this down.</p>
<center><img src="/content/images/2019/03/gokarna-hilltop.jpg" /></center>
<p>To be honest, I didn’t want the sunset to end and wanted it to just continue the whole day. And all of us just sitting there till we got the view till our heart’s content.</p>
<p>We headed back to the hostel to get a quick shower after which we got some dinner at a local restaurant which was serving <a href="https://en.wikipedia.org/wiki/Malvani_cuisine">konkani cuisine</a>.</p>
<p>Just after the dinner, we got to know about the festival which was happening at the Kotitheertha temple tank in the centre of the town and this whole section of the town was lit up with diyas and lights. Everyone was out on the streets trying to get a glimpse of the festival.</p>
<center><img src="/content/images/2019/03/Kotitheertha.jpg" /></center>
<p>The day ended with all of us crashing in our bunk beds, for me it was also the last night in Gokarna as I packed up my bags.</p>
<center><img src="/content/images/2019/03/gokarna-last-day.jpg" /></center>
<p>Morning came in quick and I had to rush to the local bus stop, to grab a bus to Gokarna beach railway station. And I thought I just cut it in time for the local train station where I was to board the train to my next stop.</p>
<h3 id="next-stop-the-historic-city-of-margao-goa">Next stop the historic city of Margao, Goa</h3>
<p>The train journey was joined by Michael, whom I had met back in the hostel where I was staying over. The train ride there was a meagre 30 INR and it would take roughly 3 hours to reach the destination.</p>
<center><img src="/content/images/2019/03/goa-train.jpg" /></center>
<p>After reaching Panjim, which was the last stop of the train. I and Michael parted ways as he was headed further north of Candolim, which was where I was going.</p>
<p>Local transport is a little bit hard to find in Goa is what I noticed, most tourists either rent a car or a scooter and you can do the whole city in that. But many are left with the option of going around with the other options.</p>
<p>I reached my hostel and freshened up a bit before heading straight to the beach right behind my hostel, it was a meagre 5-minute walk back to the beach and I was greeted by quite an old restaurant called Mama Cecilia’s shack. Ordered some shrimps and just sat down on the beachside chairs to relax and watch the sunset while dogs were running around in the water, playing with their owners and people jogging.</p>
<center><img src="/content/images/2019/03/goa-day-1.jpg" /></center>
<p>Candolim beach is another relatively busy soft sandy beach that due to erosion can be narrow in some places. Hawkers selling mostly cheap clothes and along with masseurs offering massages makes the beach a lively spot. A polite but firm no always works ~ though these people can be very interesting to talk to.</p>
<p>Hands down, the best restaurant in Candolim is After 7. Great selection of food and wine (mainly European) served in a beautiful garden. Great service, very romantic. The chap that owns the place ‘Leo’, super friendly, and very passionate about his restaurant. Opp Pedro Martina Resort, 1/274b, Gaura Vaddo.</p>
<p>Candolim itself is a small (but populated) village and it can be covered by a brief walk in 20 mins. Walking or bicycling along roads is not a great pleasure as its crowded with cars, extremely noisy and dusty. Proper walking passes yet to come.</p>
<center><img src="/content/images/2019/03/candolim.jpg" /></center>
<center><img src="/content/images/2019/03/sunset-goa.jpg" /></center>
<p>I crashed around in my bunk bed as I settled in for the night after lazing around and having conversations with the folks in the common area of the hostel, where I met a guy from China who had been living in India for some months now.</p>
<p>He mentioned that he used to work for a startup back in China and described how he had no work-life balance when he was there and got burned out. He planned to stay a few more months, travelling to Varanasi and Rajasthan before heading back.</p>
<p>The next morning was quite pleasant, woke up and had some breakfast and Mama Cecilia’s. The shrimps are to die for! I headed to <a href="https://en.wikipedia.org/wiki/Basilica_of_Bom_Jesus">Basilica de Bom Jesus</a></p>
<p>Famous throughout the Roman Catholic world, the imposing Basilica de Bom Jesus contains the tomb and mortal remains of St Francis Xavier, the so-called Apostle of the Indies. St Francis Xavier’s missionary voyages throughout the East became legendary. His ‘incorrupt’ body is in the mausoleum to the right, in a glass-sided coffin amid a shower of gilt stars.</p>
<p>Constructed between 1594 and 1605 AD, this church has a main altar, four side altars, two chapels, a sacristy and a choir.</p>
<center><img src="/content/images/2019/03/basilca.jpg" /></center>
<p>The main altar, the whole back wall is designed, like the facade in numerous carvings in wood, of pillars, friezes and arabesques all gilt in pure gold. Above the Altar and Tabernacle stands a giant statue of St. Ignatius of Loyola in the priestly vestments, nearly three meters high.</p>
<p>The church is dedicated to Jesus — in Portuguese Bom Jesus meaning Good Jesus.</p>
<p>The day ended with me heading back to the hostel and packing my bags, yet again to my final destination. Which was home indeed.</p>
<p>I boarded a bus back to Bangalore, this one being an Airavat. As I felt I really need to rest for a bit as the last few days had quite been something for me.</p>
<p>Reaching home, was another level of satisfaction. I had finally completed the trip which was an impromptu one and I was very happy that I made it.</p>
<p>I did something very similar a few years back when I did my <a href="https://www.tasdikrahman.com/2016/12/22/solo-backpacking-trip-himachal-pradesh-shimla-manali-kullu-kasol-manikaran-bhuntar-triund-dharamshala-mcleodganj-budget-india/">backpacking trip to Himachal</a>while I was in college. And this felt very similar, just that this one was solo.</p>
<p>7 days of backpacking across the southern edge of India had come to an end after completing close to 1400kms.</p>
<p>As you might have noticed, I had to rush through the last part of the trip. The initial plan was to spend two nights in every place, but as luck would have had it. It was cut short as I had to be present in Bangalore for DevOpsDays India, where I would be giving <a href="https://www.youtube.com/watch?v=3WgqFoo9eek&feature=youtu.be">a talk</a>.</p>
<p>Would love to visit the southern coast again, covering Kerala and Mangalore. Cochin and Aleppey comes over my mind.</p>
<p>Until next time!</p>
Object Comparison
2019-03-21T00:00:00+00:00
https://www.tasdikrahman.com/2019/03/21/object-equality-object-oriented-programming
<p>When do you say two objects are equal?</p>
<p>Taking example of the below two and having ruby as the language,</p>
<h3 id="comparing-primitive-objects">Comparing primitive objects</h3>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">001</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="mi">1</span> <span class="o">==</span> <span class="mi">1</span>
<span class="o">=></span> <span class="kp">true</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">002</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="s1">'tasdik'</span> <span class="o">==</span> <span class="s1">'tasdik'</span>
<span class="o">=></span> <span class="kp">true</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">003</span><span class="p">:</span><span class="mi">0</span><span class="o">></span>
</code></pre></div></div>
<h3 id="comparing-custom-objects">Comparing custom objects</h3>
<p>But what if you are having a custom class defined which has attributes for itself, you can’t really compare them using
<code class="language-plaintext highlighter-rouge">==</code> default.</p>
<p>For example.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">001</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="k">module</span> <span class="nn">Money</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">002</span><span class="p">:</span><span class="mi">1</span><span class="o">></span> <span class="k">class</span> <span class="nc">Wallet</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">003</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="nb">attr_accessor</span> <span class="ss">:rupee</span><span class="p">,</span> <span class="ss">:paise</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">004</span><span class="p">:</span><span class="mi">2</span><span class="o">></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">005</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="ss">rupee: </span><span class="mi">0</span><span class="p">,</span> <span class="ss">paise: </span><span class="mi">0</span><span class="p">)</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">006</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="vi">@rupee</span> <span class="o">=</span> <span class="n">rupee</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">007</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="vi">@paise</span> <span class="o">=</span> <span class="n">paise</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">00</span><span class="mi">8</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="k">end</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">00</span><span class="mi">9</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="k">end</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">010</span><span class="p">:</span><span class="mi">1</span><span class="o">></span> <span class="k">end</span>
<span class="o">=></span> <span class="ss">:initialize</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">011</span><span class="p">:</span><span class="mi">0</span><span class="o">></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">012</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="n">office_wallet</span> <span class="o">=</span> <span class="no">Money</span><span class="o">::</span><span class="no">Wallet</span><span class="p">.</span><span class="nf">new</span>
<span class="o">=></span> <span class="c1">#<Money::Wallet:0x00007fb4328889c8 @rupee=0, @paise=0></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">013</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="n">personal_wallet</span> <span class="o">=</span> <span class="no">Money</span><span class="o">::</span><span class="no">Wallet</span><span class="p">.</span><span class="nf">new</span>
<span class="o">=></span> <span class="c1">#<Money::Wallet:0x00007fb430991290 @rupee=0, @paise=0></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">014</span><span class="p">:</span><span class="mi">0</span><span class="o">></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">015</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="n">office_wallet</span> <span class="o">==</span> <span class="n">personal_wallet</span>
<span class="o">=></span> <span class="kp">false</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">016</span><span class="p">:</span><span class="mi">0</span><span class="o">></span>
</code></pre></div></div>
<p>But as we can see that the two objects have the same attributes, which would be rupee and paise having the same values.
They should, logically be the same thing and they should have returned <code class="language-plaintext highlighter-rouge">true</code> as a result of the <code class="language-plaintext highlighter-rouge">==</code> comparison</p>
<h3 id="object-equality-in-ruby">Object equality in ruby</h3>
<p><code class="language-plaintext highlighter-rouge">==</code> is a general comparison operator in ruby</p>
<p>At the Object level, <code class="language-plaintext highlighter-rouge">==</code> returns <code class="language-plaintext highlighter-rouge">true</code> only if obj and other are the same object. Typically, this method is overridden
in descendant classes to provide class-specific meaning.</p>
<p>This is the most common comparison, and thus the most fundamental place where you (as the author of a class) get to
decide if two objects are “equal” or not.</p>
<p>The double equals method should implement the general identity algorithm to an object, which usually means that you
should compare the object attributes and not if they are the same object in memory.</p>
<h3 id="equivalence-relation">Equivalence relation</h3>
<p>The general contract when we are overriding the == operator in ruby is that it should implement an
<a href="https://en.wikipedia.org/wiki/Equivalence_relation">equivalence relation</a>.</p>
<p>Which has the following properties.</p>
<ul>
<li>Reflexive: For any non-null reference value <code class="language-plaintext highlighter-rouge">x</code> , <code class="language-plaintext highlighter-rouge">x == x</code> and must return true .</li>
<li>Symmetric: For any non-null reference value <code class="language-plaintext highlighter-rouge">x</code> and <code class="language-plaintext highlighter-rouge">y</code>, <code class="language-plaintext highlighter-rouge">x == y</code> and <code class="language-plaintext highlighter-rouge">y == x</code> must return true .</li>
<li>Transitive: For any non-null reference value <code class="language-plaintext highlighter-rouge">x</code>, <code class="language-plaintext highlighter-rouge">y</code> and <code class="language-plaintext highlighter-rouge">z</code>, if <code class="language-plaintext highlighter-rouge">x == y</code> and <code class="language-plaintext highlighter-rouge">y == z</code> return <code class="language-plaintext highlighter-rouge">true</code> .Then <code class="language-plaintext highlighter-rouge">x == z</code>
should return <code class="language-plaintext highlighter-rouge">true</code> .</li>
<li>Consistent: For any non-null reference value <code class="language-plaintext highlighter-rouge">x</code>and <code class="language-plaintext highlighter-rouge">y</code>, multiple invocations of <code class="language-plaintext highlighter-rouge">x == y</code> must consistently return
true or consistently return <code class="language-plaintext highlighter-rouge">false</code> .</li>
<li>For any non-null reference value <code class="language-plaintext highlighter-rouge">x</code>, <code class="language-plaintext highlighter-rouge">x == nil</code> must return <code class="language-plaintext highlighter-rouge">false</code>.</li>
</ul>
<p>If the above rules are violated, your equality operator which you have overridden will behave erratically.</p>
<h3 id="overriding-eql-and-hash-methods-too">Overriding eql? and hash methods too</h3>
<p>But once, you have overridden the <code class="language-plaintext highlighter-rouge">==</code> operator, you must also override <code class="language-plaintext highlighter-rouge">eql?</code> and <code class="language-plaintext highlighter-rouge">hash</code> methods, the reason being that
even if you have never seen these methods being called anywhere. The reason is these are the methods the Hash object
is going to use to compare your object if you’re using in as a Hash key. The thing is, Hashes have to be fast to figure
out if a key is already in there and to be able to do this they just avoid comparing every single object, they just go
by “clustering” objects in groups by using the value returned by your object’s “hash” method and then, once in a
cluster, they compare the objects themselves using “eql?”.</p>
<p>Then searching for a key in a Hash, they first call “hash” in the key to figure out in which group it would be, then
they compare the key with all the other keys in the group using the “eql?” method.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">001</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="k">module</span> <span class="nn">Wealth</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">002</span><span class="p">:</span><span class="mi">1</span><span class="o">></span> <span class="k">class</span> <span class="nc">Money</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">003</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="nb">attr_accessor</span> <span class="ss">:rupee</span><span class="p">,</span> <span class="ss">:paise</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">004</span><span class="p">:</span><span class="mi">2</span><span class="o">></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">005</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="ss">rupee: </span><span class="mi">0</span><span class="p">,</span> <span class="ss">paise: </span><span class="mi">0</span><span class="p">)</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">006</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="vi">@rupee</span> <span class="o">=</span> <span class="n">rupee</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">007</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="vi">@paise</span> <span class="o">=</span> <span class="n">paise</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">00</span><span class="mi">8</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="k">end</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">00</span><span class="mi">9</span><span class="p">:</span><span class="mi">2</span><span class="o">></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">010</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="k">def</span> <span class="nf">==</span><span class="p">(</span><span class="n">other</span><span class="p">)</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">011</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="k">if</span> <span class="n">other</span><span class="p">.</span><span class="nf">nil?</span> <span class="o">||</span> <span class="o">!</span><span class="n">other</span><span class="p">.</span><span class="nf">instance_of?</span><span class="p">(</span><span class="no">Wealth</span><span class="o">::</span><span class="no">Money</span><span class="p">)</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">012</span><span class="p">:</span><span class="mi">4</span><span class="o">></span> <span class="kp">false</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">013</span><span class="p">:</span><span class="mi">4</span><span class="o">></span> <span class="k">else</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">014</span><span class="p">:</span><span class="mi">4</span><span class="o">></span> <span class="n">rupee</span> <span class="o">==</span> <span class="n">other</span><span class="p">.</span><span class="nf">rupee</span> <span class="o">&&</span> <span class="n">paise</span> <span class="o">==</span> <span class="n">other</span><span class="p">.</span><span class="nf">paise</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">015</span><span class="p">:</span><span class="mi">4</span><span class="o">></span> <span class="k">end</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">016</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="k">end</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">017</span><span class="p">:</span><span class="mi">2</span><span class="o">></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">01</span><span class="mi">8</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="k">alias</span> <span class="nb">eql?</span> <span class="o">==</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">01</span><span class="mi">9</span><span class="p">:</span><span class="mi">2</span><span class="o">*</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">020</span><span class="p">:</span><span class="mi">2</span><span class="o">*</span> <span class="k">def</span> <span class="nf">hash</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">021</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="p">[</span><span class="n">rupee</span><span class="p">,</span> <span class="n">paise</span><span class="p">].</span><span class="nf">hash</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">022</span><span class="p">:</span><span class="mi">3</span><span class="o">></span> <span class="k">end</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">02</span><span class="mi">9</span><span class="p">:</span><span class="mi">2</span><span class="o">></span> <span class="k">end</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">030</span><span class="p">:</span><span class="mi">1</span><span class="o">></span> <span class="k">end</span>
<span class="o">=></span> <span class="kp">nil</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">025</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="n">office_wallet</span> <span class="o">=</span> <span class="no">Money</span><span class="o">::</span><span class="no">Wallet</span><span class="p">.</span><span class="nf">new</span>
<span class="o">=></span> <span class="c1">#<Money::Wallet:0x00007f96c411ad20 @rupee=0, @paise=0></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">026</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="n">personal_wallet</span> <span class="o">=</span> <span class="no">Money</span><span class="o">::</span><span class="no">Wallet</span><span class="p">.</span><span class="nf">new</span>
<span class="o">=></span> <span class="c1">#<Money::Wallet:0x00007f96c40063d0 @rupee=0, @paise=0></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">027</span><span class="p">:</span><span class="mi">0</span><span class="o">></span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">02</span><span class="mi">8</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="n">office_wallet</span> <span class="o">==</span> <span class="n">personal_wallet</span>
<span class="o">=></span> <span class="kp">true</span>
<span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">02</span><span class="mi">9</span><span class="p">:</span><span class="mi">0</span><span class="o">></span>
</code></pre></div></div>
<p>Not the most elegant solution of a <code class="language-plaintext highlighter-rouge">hash</code> method, but I hope you get the idea.</p>
<h3 id="a-better-hash-method">A better hash method?</h3>
<p>The requirement is that, two objects</p>
<ul>
<li>have to be equal if they have the same hash values</li>
<li>but two objects having same hash values, may or may not be equal</li>
</ul>
<p>Which brings one to the question, can we have hash functions which never have collisions?</p>
<p>There have been discussions about whether it is possible for a function to exist, which <a href="https://crypto.stackexchange.com/questions/8765/is-there-a-hash-function-which-has-no-collisions">will never produce hash
collisions</a>.
But it’s a hard problem to solve.</p>
<p>So when you are trying to override the equals method in ruby, you should also override the hashand <code class="language-plaintext highlighter-rouge">eql?</code> method.</p>
<h3 id="object-equality-in-java">Object equality in JAVA</h3>
<p>Similar to ruby, when you are trying to implement equivalence relation in java, you have to override the</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">equals</code></li>
<li><code class="language-plaintext highlighter-rouge">hasCode</code></li>
</ul>
<p>methods in java</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kp">public</span> <span class="n">boolean</span> <span class="n">equals</span><span class="p">(</span><span class="no">Object</span> <span class="n">object</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">this</span> <span class="o">==</span> <span class="n">object</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kp">true</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">object</span> <span class="o">==</span> <span class="n">null</span> <span class="o">||</span> <span class="o">**</span><span class="n">getClass</span><span class="p">()</span> <span class="o">!=</span> <span class="n">object</span><span class="p">.</span><span class="nf">getClass</span><span class="p">()</span><span class="o">**</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kp">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="no">MyClass</span> <span class="n">other</span> <span class="o">=</span> <span class="p">(</span><span class="no">MyClass</span><span class="p">)</span> <span class="n">object</span><span class="p">;</span>
<span class="k">return</span> <span class="n">this</span><span class="p">.</span><span class="nf">x</span> <span class="o">==</span> <span class="n">other</span><span class="p">.</span><span class="nf">x</span> <span class="o">&&</span> <span class="n">this</span><span class="p">.</span><span class="nf">y</span> <span class="o">==</span> <span class="n">other</span><span class="p">.</span><span class="nf">y</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>One of the most common mistakes which people make while overriding the equals method is that instead of using the
<code class="language-plaintext highlighter-rouge">Object</code> class, they use the class, in which they are overriding the equals method itself.</p>
<h3 id="references">References</h3>
<ul>
<li>https://www.harukizaemon.com/blog/2005/12/28/how-to-write-eql-in-ruby/</li>
<li>https://ruby-doc.org/core-2.5.3/Hash.html#class-Hash-label-Hash+Keys</li>
<li>https://javarevisited.blogspot.com/2011/02/how-to-write-equals-method-in-java.html</li>
<li>https://stackoverflow.com/questions/7156955/whats-the-difference-between-equal-eql-and</li>
<li>https://stackoverflow.com/questions/1931604/whats-the-right-way-to-implement-equality-in-ruby’</li>
<li>https://mauricio.github.io/2011/05/30/ruby-basics-equality-operators-ruby.html</li>
</ul>
What should and should not be tested in unit tests?
2019-03-13T00:00:00+00:00
https://www.tasdikrahman.com/2019/03/13/what-should-and-should-not-be-tested-in-unit-tests
<blockquote>
<p>I have written about <a href="https://www.tasdikrahman.com/2019/03/13/f-i-r-s-t-principles-of-testing/">F.I.R.S.T principles</a> of testing and <a href="https://www.tasdikrahman.com/2019/02/08/why-should-I-follow-test-driven-development/">TDD as a school of thought</a></p>
</blockquote>
<p>Probably an extreme opinion, but this is how <a href="https://blog.codinghorror.com/i-pity-the-fool-who-doesnt-write-unit-tests/">Jeff Atwood</a> puts it</p>
<blockquote>
<p>I Pity The Fool Who Doesn’t Write Unit Tests</p>
</blockquote>
<h3 id="but-what-should-you-test">But what should you test?</h3>
<p>This I generally try to follow</p>
<ul>
<li>Test the common case of everything you can. This will tell you when that code breaks after you make some change (which is, in my opinion, the single greatest benefit of automated unit testing).</li>
<li>Test the edge cases of a few unusually complex code that you think will probably have errors.</li>
<li>Whenever you find a bug, write a test case to cover it before fixing it</li>
<li>Add edge-case tests to less critical code whenever someone has time to kill.</li>
</ul>
<p>This will not only help you deliver and release faster, but will also make you more confident about your own codebase.</p>
<blockquote>
<p>Writing tests and having 100% code coverage does not necessarily mean that your code is bug free, but I feel it’s certainly better than having no tests at all.</p>
</blockquote>
<h3 id="what-should-you-not-test">What should you not test?</h3>
<ul>
<li>The code is trivial. A getter that returns 0 doesn’t need to be tested, and changes will be covered by tests for its consumers.</li>
<li>The code simply passes through into a stable API. I’ll assume that the standard library works properly.</li>
<li>The code needs to interact with other deployed systems; then an integration test is called for.</li>
<li>If the test of success/fail is something that is so difficult to quantify as to not be reliably measurable, such as steganography being unnoticeable to humans.</li>
<li>If the test itself is an order of magnitude more difficult to write than the code.</li>
<li>If the code is throw-away or placeholder code. If there’s any doubt, test.</li>
</ul>
<h3 id="references">References</h3>
<ul>
<li>https://softwareengineering.stackexchange.com/a/147075/169827</li>
<li>https://softwareengineering.stackexchange.com/a/754/169827</li>
<li>https://blog.codinghorror.com/i-pity-the-fool-who-doesnt-write-unit-tests/</li>
</ul>
F.I.R.S.T principles of testing
2019-03-13T00:00:00+00:00
https://www.tasdikrahman.com/2019/03/13/f-i-r-s-t-principles-of-testing
<h1 id="first-principles-of-testing-stand-for">First principles of testing stand for</h1>
<ul>
<li>Fast</li>
<li>Isolated/Independent</li>
<li>Repeatable</li>
<li>Self-validating</li>
<li>thorough</li>
</ul>
<blockquote>
<p>Bugs are introduced in the parts of code, which we usually don’t pay attention to, or places which are too hard to understand.</p>
</blockquote>
<h3 id="fast">Fast</h3>
<p>The developer shouldn’t hesitate to run the run the unit tests at any point of their development cycle, even if there are thousands of unit tests. They should run and show you the desired output in a matter of seconds</p>
<h3 id="isolated">Isolated</h3>
<p>For any given unit test, for its environment variables or for its setup. It should be independent of everything else should so that it results is not influenced by any other factor.</p>
<p>Should follow the <a href="https://xp123.com/articles/3a-arrange-act-assert/">3 A’s of testing: Arrange, Act, Assert</a></p>
<p>In some literature, it’s also called as <a href="https://martinfowler.com/bliki/GivenWhenThen.html">Given, when, then</a>.</p>
<h3 id="arrange">Arrange</h3>
<p>All the data should be provided to the test when you’re about to run the test and it should not depend on the environment you are running the tests</p>
<h3 id="act">Act</h3>
<p>Invoke the actual method under test</p>
<h3 id="assert">Assert</h3>
<p>At any given point, a unit test should only assert one logical outcome, multiple physical asserts can be part of this physical assert, as long as they all act on the state of the same object.</p>
<p>Preferably, don’t do any actions after the assert call</p>
<h3 id="repeatable">Repeatable</h3>
<p>Tests should be repeatable and deterministic, their values shouldn’t change based on being run on different environments.
Each test should set up its own data and should not depend on any external factors to run its test</p>
<h3 id="self-validating">Self-validating</h3>
<p>You shouldn’t need to check manually, whether the test passed or not.</p>
<h3 id="thorough">Thorough</h3>
<ul>
<li>should cover all the happy paths</li>
<li>try covering all the edge cases, where the author would feel the function would fail.</li>
<li>test for illegal arguments and variables.</li>
<li>test for security and other issues</li>
<li>test for large values, what would a large input do their program.</li>
<li>should try to cover every use case scenario and not just aim for 100% code coverage.</li>
</ul>
<h2 id="references">References:</h2>
<ul>
<li>https://github.com/ghsukumar/SFDC_Best_Practices/wiki/F.I.R.S.T-Principles-of-Unit-Testing</li>
<li>https://martinfowler.com/bliki/GivenWhenThen.html</li>
<li>https://xp123.com/articles/3a-arrange-act-assert/</li>
</ul>
Test-driven development as a school of thought
2019-02-08T00:00:00+00:00
https://www.tasdikrahman.com/2019/02/08/why-should-I-follow-test-driven-development
<p><a href="https://www.wsj.com/articles/SB10001424053111903480904576512250915629460">Software is eating the world</a>, and so is the world of software development constantly changing.</p>
<p>One way of developing a project would involve</p>
<ul>
<li>analysts figuring out the business requirements and sit for a few weeks if not months</li>
<li>these requirements would be given out to the architects who would in case break the problem down into manageable chunks</li>
<li>the chunks themselves would be then given out to the teams which would be the delivering the specific modules.</li>
</ul>
<p>Nothing, but a typical <a href="https://en.wikipedia.org/wiki/Waterfall_model">waterfall model</a> scenario, coming with it’s obvious pros and cons.</p>
<p>TDD or Test driven development follows a similar approach to this, which would be gathering the requirements, doing the analysis, modelling the desired system before writing any code for it.</p>
<p>To write a test, you will be gathering what the input should be and what the output should be(on a very high level of thought).</p>
<p>Once you have some knowledge about the what needs to be done, you will be able to write those fine grained requirements in the form of tests.</p>
<p>When you are writing those tests, you will figure out the gaps and misunderstandings in the requirements before you’ve committed those gaps and misunderstandings to the project in the form of executable code.</p>
<p>What you would gain immediately after adopting TDD</p>
<ul>
<li>you get a better understanding of the project that you’re gonna write.</li>
</ul>
<p>This is because of the fact that, you are only implementing the features that are required immediately in the first iteration.</p>
<ul>
<li>you will have more confidence while refactoring your code</li>
</ul>
<p>The problem comes when you would want to refactor certain areas of your code base, and this is where TDD shines if you ask me. You would be confident about the functionality as you would only be testing the end behaviour of your program and not the implementation details of how is it being achieved in unit tests.</p>
<p>This way, you can easily go with the <a href="https://blog.cleancoder.com/uncle-bob/2014/12/17/TheCyclesOfTDD.html">Red Green refactor</a> cycle.</p>
<ul>
<li>immediate feedback on whether what you’re working is fine or not, hence fewer bugs</li>
</ul>
<p>If you’re covering all the edge cases as part of writing the tests, you are already sorted in having a system which has minimal bugs in terms of functionality.</p>
<ul>
<li>loosely coupled, highly modular code.</li>
</ul>
<p>Something which I have been trying to follow lately is to not check in any production code, before writing the tests for it.</p>
<p>Write the specs, run the spec, let it fail (red). Implement the interface that you’re testing (make the code green), and then refactor.</p>
<p>Some of the things which I noticed worked well while writing tests were</p>
<ul>
<li>Avoid writing procedural style tests</li>
<li>Use test doubles while testing, and make sure you are running single units of code in isolation.</li>
<li>Follow the <a href="https://martinfowler.com/bliki/GivenWhenThen.html">given when then</a> style of formatting your tests</li>
<li>Unit tests need true isolation, and they shouldn’t be hitting databases or opening sockets when you are testing something.</li>
<li>Use only one assertion per test.</li>
</ul>
<p>Testing is like security, you can never be 100% per cent sure whether you’ve got it, but it surely adds to the confidence on what you’ve built.</p>
<p>So let’s say if you have written some 20 tests, and all of them pass, you wouldn’t be sure if what you have written is correct or not. But let’s say you started with all them being in the red state, you would have much more confidence in the system that you have built.</p>
<p>This back and forth does take time. But it’s a process which tries to address the concern of having doubts about whether what you built is a resilient system or not.</p>
<h2 id="resources">Resources</h2>
<ul>
<li>http://misko.hevery.com/2008/08/14/procedural-language-eliminated-gotos-oo-eliminated-ifs/</li>
<li>https://martinfowler.com/bliki/GivenWhenThen.html</li>
<li>https://stackoverflow.com/questions/920992/unit-test-adoption</li>
<li>http://www.natpryce.com/articles/000714.html</li>
<li>https://javaranch.com/journal/200603/EvilUnitTests.html</li>
</ul>
Moving Canary deployments on AWS using ELB to kubernetes using Traefik
2018-10-25T00:00:00+00:00
https://www.tasdikrahman.com/2018/10/25/canary-deployments-on-AWS-and-kubernetes-using-traefik
<p><a href="https://martinfowler.com/bliki/CanaryRelease.html">Canary deployment</a> pattern is very similar to <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">Blue green deployments</a>, where you are deploying a certain version of your application to a subset of your application servers. If everything is alright and you have tested out that everything is working fine, you route a certain percentage of your users to those application servers and gradually keep increasing the traffic till a full rollout is achieved.</p>
<p>One of the many reasons to do this can be to test a certain feature out with a percentage of users who use your service. This can be further extended to enabling a service to users of a particular demographic.</p>
<h2 id="canary-deployments-on-aws">Canary deployments on AWS</h2>
<p>Canary in our use case @ <a href="https://www.razorpay.com">Razorpay</a>, was used by one of our API’s which we served and gave out for consumption and the earlier method for canary deployments there before we were on <a href="https://kubernetes.io">kubernetes</a> was to have two seperate <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html">Autoscaling Groups</a> for the primary ASG serving the particular API and another ASG(with a smaller desired count for the ASG), let’s call it canary ASG for now.</p>
<p>Now both</p>
<ul>
<li>primary ASG</li>
<li>canary ASG</li>
</ul>
<p>were having their own individual <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html">ELB’s</a> attached to them, with them being in our public subnet. Both the ELB’s would have a CNAME DNS record pointing to their public FQDN given out by AWS.</p>
<center><img src="/content/images/2018/10/vpc-diagram.png" /></center>
<p>For simplicity of drawing the ASG groups, I have not shown the ASG groups for both the canary and the main service in 2 separate AZ’s, but it is the recommended way to go forward. As in case of an AZ failure, you have the other set of ASG instances to be routed by the ELB(with cross zone load balancing enabled)</p>
<p>The canary ASG would be attached to both the</p>
<ul>
<li>main ELB for the service</li>
<li>canary’s separate ELB</li>
</ul>
<p>The capacity(min: desired) for the main service is more than the capacity for the ASG for canary, and the canary ASG capacity’s max is set to it’s desired. The reasoning for this is that, any regression wouldn’t propagate to a larger number of users if autoscaling kicks in.</p>
<p>Since our ELB is an Internet-facing load balancer, it gets public IP addresses(one for each AZ). The DNS name of an Internet-facing load balancer is publicly resolvable to the public IP addresses of the nodes of the ELB. Therefore, Internet-facing load balancers can route requests from clients over the Internet.</p>
<p>The load balancer node that receives the request selects an attached instance using the <a href="https://community.cisco.com/t5/routing/what-is-round-robin-routing/td-p/1400406">round robin routing</a> algorithm for TCP listeners and the least outstanding requests routing algorithm for HTTP and HTTPS listeners.</p>
<p>Hence, the canary instances would also get traffic in a round robin fashion.</p>
<h2 id="replicating-the-same-in-kubernetes">Replicating the same in kubernetes</h2>
<p><a href="https://traefik.io/">traefik</a> runs as our L7 load balancer, or as our <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-controllers">ingress controller</a> inside kubernetes to route traffic to our <a href="https://kubernetes.io/docs/concepts/services-networking/service/">kubernetes services</a> for various microservices running inside our cluster.</p>
<p>traefik would be running on <code class="language-plaintext highlighter-rouge">hostNetwork: true</code> as <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">DaemonSet</a></p>
<p>These pods will use the host network directly and not the “<a href="https://kubernetes.io/docs/concepts/cluster-administration/networking/">pod network</a>” (the term “pod network” is a little bit misleading as there is no such thing - it basically just comes down to routing network packets and namespaces). So we can bind the Traefik ports on the host interface on port <code class="language-plaintext highlighter-rouge">80</code>. That also means of course that no further pods of a DaemonSet can use this port and of course also no other services on the worker nodes. But that’s what we want here as Traefik is basically our “external” loadbalancer for our “internal” services - our tunnel to the rest of the internet so to say.</p>
<p>Sample configuration which you can use to deploy traefik</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRoleBinding</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-controller</span>
<span class="na">roleRef</span><span class="pi">:</span>
<span class="na">apiGroup</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-controller</span>
<span class="na">subjects</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-controller</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-controller</span>
<span class="na">rules</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">pods</span>
<span class="pi">-</span> <span class="s">services</span>
<span class="pi">-</span> <span class="s">endpoints</span>
<span class="pi">-</span> <span class="s">secrets</span>
<span class="na">verbs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">get</span>
<span class="pi">-</span> <span class="s">list</span>
<span class="pi">-</span> <span class="s">watch</span>
<span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">extensions</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ingresses</span>
<span class="na">verbs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">get</span>
<span class="pi">-</span> <span class="s">list</span>
<span class="pi">-</span> <span class="s">watch</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">ConfigMap</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span>
<span class="na">data</span><span class="pi">:</span>
<span class="na">traefik-config</span><span class="pi">:</span> <span class="pi">|-</span>
<span class="s">defaultEntryPoints = ["http","https"]</span>
<span class="s">[entryPoints]</span>
<span class="s">[entryPoints.http]</span>
<span class="s">address = ":80"</span>
<span class="s">[entryPoints.http.redirect]</span>
<span class="s">regex = "^http://(.*)"</span>
<span class="s">replacement = "https://$1"</span>
<span class="s">[entryPoints.https]</span>
<span class="s">address = ":443"</span>
<span class="s">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-controller</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-service</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">traefik-ingress-lb</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">80</span>
<span class="pi">-</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">admin</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">8080</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">NodePort</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">DaemonSet</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-controller</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">traefik</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">traefik-ingress-lb</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">matchLabels</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">traefik-ingress-lb</span>
<span class="na">updateStrategy</span><span class="pi">:</span>
<span class="na">rollingUpdate</span><span class="pi">:</span>
<span class="na">maxUnavailable</span><span class="pi">:</span> <span class="m">1</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">k8s-app</span><span class="pi">:</span> <span class="s">traefik-ingress-lb</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-lb</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">nodeSelector</span><span class="pi">:</span>
<span class="na">edge-node-label</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">serviceAccountName</span><span class="pi">:</span> <span class="s">traefik-ingress-controller</span>
<span class="na">terminationGracePeriodSeconds</span><span class="pi">:</span> <span class="m">60</span>
<span class="na">hostNetwork</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">image</span><span class="pi">:</span> <span class="s">traefik:v1.7.16-alpine</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik-ingress-lb</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
<span class="na">containerPort</span><span class="pi">:</span> <span class="m">80</span>
<span class="na">hostPort</span><span class="pi">:</span> <span class="m">80</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">admin</span>
<span class="na">containerPort</span><span class="pi">:</span> <span class="m">8080</span>
<span class="na">securityContext</span><span class="pi">:</span>
<span class="na">privileged</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">args</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">--loglevel=INFO</span>
<span class="pi">-</span> <span class="s">--web</span>
<span class="pi">-</span> <span class="s">--kubernetes</span>
<span class="pi">-</span> <span class="s">--web.metrics.prometheus</span>
<span class="pi">-</span> <span class="s">--web.metrics.prometheus.buckets=0.1,0.3,1.2,5</span>
<span class="pi">-</span> <span class="s">--configFile=/etc/traefik/traefik.toml</span>
<span class="na">resources</span><span class="pi">:</span>
<span class="na">limits</span><span class="pi">:</span>
<span class="na">cpu</span><span class="pi">:</span> <span class="s">200m</span>
<span class="na">memory</span><span class="pi">:</span> <span class="s">300Mi</span>
<span class="na">requests</span><span class="pi">:</span>
<span class="na">cpu</span><span class="pi">:</span> <span class="s">100m</span>
<span class="na">memory</span><span class="pi">:</span> <span class="s">150Mi</span>
<span class="na">volumeMounts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">config-volume</span>
<span class="na">mountPath</span><span class="pi">:</span> <span class="s">/etc/traefik</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">config-volume</span>
<span class="na">configMap</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">traefik</span>
<span class="na">items</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">traefik-config</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">traefik.toml</span>
</code></pre></div></div>
<center><img src="/content/images/2018/10/canary-k8s-traefik-vpc.png" /></center>
<p>The diagram above shows two ASG’s for edge nodes, which will host the traefik daemonset(s).</p>
<p>There would be a CNAME DNS record for <code class="language-plaintext highlighter-rouge">myapp.example.com</code> which points to the public FQDN for the common ELB to which both the edge ASG’s are attached to. Traffic would be routed to the edge VM’s attached based on a round robin fashion here. Before that, the security groups attached to the ASG’s can also be configured to only allow TCP connections on port 80(others would be blocked automatically as it’s default deny).</p>
<p>Similarly a DNS record for canary would be there.</p>
<p>Traefik would be listening on port 80 on the host’s network for incoming requests, and there would be an ingress object in the namespace of the app which would define which service to route the traffic based on the hostname.</p>
<p>We can have an ingress object like the following in the namespace <code class="language-plaintext highlighter-rouge">myapp</code> for the services</p>
<ul>
<li>myapp</li>
<li>myapp-canary</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># this feature is available only from traefik version 1.7.0 and upwards
# https://github.com/containous/traefik/releases/tag/v1.7.0
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: traefik-external
traefik.ingress.kubernetes.io/service-weights: |
myapp: 90%
myapp-canary: 10%
name: myapp-ingress
namespace: myapp
spec:
rules:
- host: myapp.example.com
http:
paths:
- backend:
serviceName: myapp
servicePort: 80
path: /
- backend:
serviceName: myapp-canary
servicePort: 80
path: /
status:
loadBalancer: {}
</code></pre></div></div>
<p>This way, traefik would route the traffic coming to <code class="language-plaintext highlighter-rouge">myapp.example.com</code> to the services</p>
<ul>
<li>myapp : 90% of the traffic would be routed here.</li>
<li>myapp-canary: 10% of the traffic would be routed here.</li>
</ul>
<p>You can have multiple services to which you can specify the weights and I would only be repeating myself to what has been written here https://docs.traefik.io/user-guide/kubernetes/#traffic-splitting</p>
<p>Another thing to note here is that, the service to which you are trying to do a weighted routing for your canary, should be in the same namespace as the other service(which is <code class="language-plaintext highlighter-rouge">myapp</code> in this case). This was asked here in their issue tracker and they pointed out the same https://github.com/containous/traefik/issues/4043.</p>
<p>So you have seen how we can do canary deployments on AWS using traditional ELB’s and ASG’s as well as if you are on kubernetes with an ingress controller.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html">https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html</a></li>
<li><a href="https://martinfowler.com/bliki/CanaryRelease.html">https://martinfowler.com/bliki/CanaryRelease.html</a></li>
<li><a href="https://github.com/containous/traefik/pull/3112">https://github.com/containous/traefik/pull/3112</a> PR where weighted traffic was added to traefik</li>
<li><a href="https://docs.traefik.io/configuration/backends/kubernetes/">https://docs.traefik.io/configuration/backends/kubernetes/</a></li>
<li><a href="https://docs.traefik.io/user-guide/kubernetes/#traffic-splitting">https://docs.traefik.io/user-guide/kubernetes/#traffic-splitting</a></li>
</ul>
<h2 id="credits">Credits</h2>
<p>The Network diagrams were made using <a href="https://draw.io">draw.io</a></p>
Monoliths are just fine
2018-10-24T00:00:00+00:00
https://www.tasdikrahman.com/2018/10/24/monoliths-are-just-fine
<p>A lot of great material has already been <a href="https://martinfowler.com/articles/microservices.html">written</a> out there around what microservices are and what they are not.</p>
<p>What I would try putting down here is what I saw as we grew from a monolith to a microservices architecture over the period of time back here at <a href="https://www.razorpay.com">Razorpay</a>. Please take it with a grain of salt when you read this as this is going to be opinionated.</p>
<blockquote>
<p>Microservices are something which you grow into, not something you start with</p>
</blockquote>
<p>Having a microservice(s) based architecture just for the sake of having one is kind of like using <a href="https://kubernetes.io">kubernetes</a> to deploy a service or two on it. In the end, you will only overcomplicate things by having multiple things to manage. And trust me, <a href="https://codeengineered.com/blog/2017/kubernetes-is-hard/">kubernetes is hard</a>. I say that with at least a year and a half of running production workloads at Razorpay on self-hosted kubernetes and having scaled them to what it is now. It’s complicated and can be hard to get right at first. But makes life much easier when you have multiple services to manage at the very least.</p>
<p>If you have a look at Shopify, they are one huge Ruby on Rails shop. And the last time I checked their scale is pretty huge. So the argument that microservices are the only way to scale is not entirely true.</p>
<p>There are reasons why I feel monoliths are the way one should start with, because</p>
<blockquote>
<p>Microservices are complicated to manage</p>
</blockquote>
<p>If you’re just starting to write your app, you need to get market validation fast and get the features out before your competition does, instead of having to worry about 10 other different things. Your customers/investors probably won’t care about how your service works or what is their underlying tooling here unless they are techies and would be curious about how is it working internally but at the end of the day, they just want things to work!</p>
<p>This is the part, where you are working on your prototype and you don’t need the extra overhead of worrying about how the 10 different microservices you wrote are interacting with each other. As at the end of the day, the tooling and the architecture matters, but not if you have not even attained that critical mass. This is a little similar to fight over what language to choose over the other, of course, there would be obvious choices for some specific jobs but you can always go first with the one which you and the rest of the team are most comfortable with to <a href="http://www.paulgraham.com/avg.html">beat the averages</a></p>
<p>Plus the overhead of managing the health checks, monitoring and graceful shutdown for all the microservices is something which you need to put on your head around when starting with microservices apart from writing your main business logic.</p>
<p>They bring a lot of baggage that you might never have seen before (i.e. big learning curve) and the myth of isolated changes is just that, a myth. Unless it is some low-level thing, you cannot change it without impacting other services and this is no different than a monolith. It just replaces internal calls between services of your monolith with flaky and slower network calls.</p>
<blockquote>
<p>Architect your monolith right</p>
</blockquote>
<p>When you start off with your monolith, there would be places where you feel you see things repeating. And that’s where you separate that out into functions or modules. But you should not be afraid to repeat yourself at the start and try to modularise/abstract out everything, as you would not like to end up with <a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">leaky abstractions</a></p>
<p>When you understand the boundaries and functions well enough, that’s when you would start modularizing your codebase. So as to have clear distinctions of which part of the codebase does what. This helps in a few things, specific teams being able to work on specific codebases and clear distinction between core functionality and helper functions.</p>
<p>When you are done dividing your codebase into modules, using a message queue for asynchronous communication between the modules would make more sense. This will enable you to debug faster, distribute the work of different components to different teams for things to start with.</p>
<p>With all the above in check, you already have a lot of things sorted which would not become a technical debt when trying to move to microservices.</p>
<p>From what I have noticed is, over-optimising from the start is just gonna bog you down. The priority at the start should be to</p>
<ul>
<li>Get it working.</li>
<li>Get feedback about your product.</li>
<li>Fix the bugs when they are reported or when you find any.</li>
<li>If things go down, look why they went wrong and fix them. This is where you will learn from your experience.</li>
<li>Repeat the whole process.</li>
</ul>
<p>I think of it this way, your product is not a program like <a href="http://man7.org/linux/man-pages/man1/ls.1.html">ls</a> command which is feature complete, you need to constantly iterate upon it, but even before that. You have to give out a working model for people out there to use it.</p>
<blockquote>
<p>Moving to a microservice?</p>
</blockquote>
<p>The argument of moving to a microservice can be made when</p>
<ul>
<li>you have divided certain work among certain teams, by virtue of which there would be times when there would be friction, miscommunication happening over when contributing to certain parts of the codebase which come common when different teams are working. Even if you have managed to write something, it’s quite possible that something which you added might have had a regression over something, and at this point, your automated tests in the CI should ideally catch them. But having a clear separation of work when you have multiple teams working on things, microservices can make sense.</li>
<li>you would want to rewrite a piece of codebase into something more performant.</li>
<li>Feature Velocity</li>
<li>autonomy of teams to iterate faster with their own choice of technology</li>
<li>make deployments quicker, impact on story structure.</li>
</ul>
<p>The jump from monolith to service-oriented thinking is a huge one. But the jump from a few services to more is much easier.</p>
<p>Most of the times, I’ve found a push to microservices within an organization to be due to some combination of:</p>
<p>1) Business is pressuring tech teams to deliver faster, and they cannot, so they blame current system (derogatory name: monolith) and present microservices as solution. Note, this is the same tired argument from years ago when people would refer to legacy systems/legacy code as the reason for not being able to deliver.</p>
<p>2) Inexperienced developers proposing microservices because they think it sounds much more fun than working on the system as it is currently designed.</p>
<p>3) Technical people trying to avoid addressing the lack of communication and leadership in the organization by implementing technical solutions. This is common in the case where tech teams end up trying to “do microservices” as a way to reduce merge conflicts or other such difficulties that are ultimately a problem of human interaction and lack of leadership. Technology does not solve these problems.</p>
<p>4) Inexperienced developers not understanding the immense costs of coordination/operations/administration that come along with a microservices architecture.</p>
<p>5) Some people read about microservices on the engineering blog of one of the major tech companies, and those people are unaware that such blogs are a recruiting tool of said company. Many (most?) of those posts are specifically designed to be interesting and present the company as doing groundbreaking stuff in order to increase inbound applicant demand and fill seats. Those posts should not be construed as architectural advice or <em>best practices</em></p>
<p>In the end, it’s absolutely the case that a movement to microservices is something that should be evolutionary, and in direct need to technical requirements. For nearly every company out there, a horizontally-scaled monolith will be much simpler to maintain and extend than some web of services, each of which can be horizontally scaled on their own.</p>
<p>IMHO, the monoliths vs microservices debate is akin to monorepos vs multi-repos: they are both strategies used to share work when your organization grows. Both can work well, depending on your tooling and organization.</p>
<p>But do not forget that those abstractions layers you add, while very useful (say, for release velocity), might also be a direct application of <a href="https://en.wikipedia.org/wiki/Conway%27s_law">Conway’s Law</a></p>
<p>Which means that refactoring some code, might sometime require refactoring your organisation, so if you lack the ability to do that incrementally, you might converge to an ossified system that stops evolving.</p>
<blockquote>
<p>So, Microservices or Monoliths?</p>
</blockquote>
<p>It depends.</p>
<h2 id="references">References</h2>
<ul>
<li>https://martinfowler.com/bliki/MonolithFirst.html</li>
<li>https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/</li>
<li>http://www.paulgraham.com/avg.html</li>
</ul>
Pillars of Observability
2018-10-01T00:00:00+00:00
https://www.tasdikrahman.com/2018/10/01/pillars-of-observability
<p>Haven’t written around for much of this year, hope it changes going down to the end of this year. This year has been very fruitful in terms of learnings and I can’t wait to share what I have learned.</p>
<p>This post would basically be an introduction to what I have understood by the term of observability into your infrastructure and the services which are hosted on top of it.</p>
<p>There are 3 pillars of observability:</p>
<ul>
<li>
<p>metrics</p>
</li>
<li>
<p>logs</p>
</li>
<li>
<p>tracing</p>
</li>
</ul>
<p>While all of them partially overlap, each have different purpose.</p>
<h3 id="metrics">Metrics</h3>
<p>These are numbers related amount of events that happened in the time range. Like:</p>
<ul>
<li>number of successful/failed/overall requests</li>
<li>cumulative duration of requests</li>
<li>bucketed histogram of requests’ durations</li>
</ul>
<p>The main point of metrics is that these are small, you can gather them cheaply and store them for a long period of time. These give you an overall overview of the whole system, but without insights.</p>
<p>So, metrics answer to the question “how is my system performance changes through time?”.</p>
<p>I would add visualization to it too, as it goes hand in hand with metrics.</p>
<p>Historically people have used statsd along with graphite as the storage backend. I personally prefer <a href="https://prometheus.io/">prometheus</a>. Which is an open-source, metrics-based monitoring system. It does one thing and does it pretty well, with a simple yet powerful data model and a query language which lets you analyse how your applications and infrastructure are performing.</p>
<p>I believe in the philosophy that</p>
<blockquote>
<p>if something moves, you should track it and graph it. If it decides to make a run for it.</p>
</blockquote>
<p>The thing about prometheus is that they are pretty mature in the ecosystem and have a lot of support in terms of contributors from the OSS community as well as native <a href="https://kubernetes.io">kubernetes</a> metrics support. Aggregrating metrics from k8s objects is pretty straightforward with the service discovery provided by prometheus.</p>
<p>And they have a lot of exporters which they have written and some being supported by the community. And if one wants to instrument their code, they can use any of the <a href="https://prometheus.io/docs/instrumenting/clientlibs/">client libraries</a> to do so. Network metrics, application metrics, you name it.</p>
<p>Heck, a friend of mine wrote an <a href="https://git.captnemo.in/nemo/prometheus-act-exporter">exporter for an ISP provider</a>, you can actually go ahead and check the <a href="https://grafana.bb8.fun/d/_u2-GHSik/main-dashboard?refresh=5s&orgId=1">metrics here</a></p>
<p><a href="https://grafana.com/">grafana</a> can be used to visualize the data which is being scraped by prometheus (which can be added as a source to grafana), which works on a pull model.</p>
<p><a href="https://www.influxdata.com/">InfluxDB</a> is another Time series database, which would also work on a similar manner and they would have an enterprise plan for HA too, which prometheus achieves currently using <a href="https://github.com/improbable-eng/thanos">thanos</a></p>
<h3 id="logs">Logs</h3>
<p>These are single events that happened in the system, ex. single request. It will often contain also some information that is in the metrics (like request duration), but it will also contain more context, like IP of the requester and precise time of the request. The problem with logs isn’t that these are larger in size than just metrics itself, so you cannot store them for so long as the metrics and due to their size, you need to reduce the number of logs you send.</p>
<p>So, logs answers to the question “what happened in my system?”.</p>
<p>You can either go with a hosted solution or a SaaS-based solution for logging.</p>
<p>Logs are a critical part of the infra and without logs, I would say developers are just shooting in the dark when trying to debug their applications. Which makes it mission critical. I personally have experienced that going with a hosted SaaS solution, if you have a small team is something which I would suggest. As when your volumes grow, you would have to invest a lot of time to invest in fixing stuff when it breaks in the logging side. The most popular approach is pushing your logs to Elasticsearch, fluentd pushing it and kibana as the frontend. All making it stand as <a href="https://docs.fluentd.org/v0.12/articles/docker-logging-efk-compose">EFK</a>, which is actually a pretty decent way to go about it.</p>
<p>But managing your ES clusters can get nasty if you are hosting them on a public cloud. Spikes on CPU during ingestion of logs is pretty common from what I have noticed. Again, depends on whether you would want to tradeoff developer time trying to fix it or move it to a SaaS based product.</p>
<p>If you are operating at a very large scale, it would obviously make more sense for you to host logging infra inside your infrastructure as the costs would be enormous with a SaaS product.</p>
<p>Facebook recently Opensourced their distributed logging platform, <a href="https://logdevice.io/">logdevice</a>, last time I checked. One of their engineers mentioned in an <a href="https://news.ycombinator.com/item?id=17976930">HN comment</a> that</p>
<blockquote>
<p>LogDevice ingests over 1TB/s of uncompressed data at Facebook. The maximum limit as defined by default in the code for the number of storage nodes in a cluster is 512.</p>
</blockquote>
<p>I guess you get what I am trying to put across here as a point.</p>
<h3 id="tracing">Tracing</h3>
<p>This one is for monitoring how a single event is behaving in the system. So you have information on how long this request spent in LB, backend, DB, how long it spent on external services, and so on. So as you see, this overlap with both metrics and logs, but still not exactly. The main problem is that while metrics and logs can have different formats in each of the services, tracing needs to be uniform because your system needs to be “request ID aware”. Also, traces can be very large, that is why in general you do not trace all requests, but only some of them to reduce internal traffic.</p>
<p>So, tracing answers to the question “how is my system components interacting with each other?”.</p>
<p><a href="http://opentracing.io/">Opentracing</a> is just a standard that needs services like <a href="https://www.jaegertracing.io/docs/1.6/">Jaeger</a> to implement the actual tracing. Tracing tools are about following a request as it moves between different systems.</p>
<p>Will be writing another post in and around metrics with prometheus shortly, Until next time!</p>
<h2 id="references">References</h2>
<ul>
<li>http://code.flickr.com/blog/2008/10/27/counting-timing/</li>
<li>https://codeascraft.com/2011/02/15/measure-anything-measure-everything/</li>
</ul>
Trip to Taiwan, 2017
2017-10-07T00:00:00+00:00
https://www.tasdikrahman.com/2017/10/07/Trip-to-Taiwan-2017-Pycon-Taiwan-2017
<p>Dum Chai had become one of my favorites, thanks to the after work-hour chill time with colleagues.
That day being no different, we were in our usual routine.</p>
<p>But I was in for a surprise. I was getting my VISA finally after much hassles that day. All the mindless haggling and
tireless procedures with the paperwork which I had to run through at the last moment. Oh man, that was something.</p>
<center><img src="/content/images/2017/09/taiwan_trip_1.jpg" /></center>
<h2 id="exactly-a-week-later">Exactly A week later</h2>
<p>It was the first week of June and I was already in my last month of my internship at Cisco, Bangalore. My mom for a change
that day was not telling me to pack my bags (which I hadn’t till the last night before travel).</p>
<center><img src="/content/images/2017/09/taiwan_trip_clothes.jpg" /></center>
<p>My flight was quite late at night, so I had time to devour some good dinner and chill with some good music, which I did.</p>
<center><img src="/content/images/2017/09/taiwan_trip_2.jpg" /></center>
<p>Boredom was nothing to be worried about as I had Joyce by my side during the whole time. She works for a firm based out of
China and frequents a lot to Indian cities. And we had discussions ranging from the <a href="https://en.wikipedia.org/wiki/Great_Firewall">Great firewall</a>, the various
provinces of China and the cultural differences and similarities that we had. It was pretty interesting and we exchanged emails to which
she would promise me that she would be my guide when I visit China the next time.</p>
<center><img src="/content/images/2017/09/taiwan_trip_3.jpg" /></center>
<p>I had a transit via Bangkok which I reached quite early that morning. The layover got over quickly while I changed my terminal
to catch my connecting flight to Taipei.</p>
<p>It was just the time of sunrise and trust me, even if I was sleep deprived. The sunrise was quite something. Dunno if I did justice
while capturing it.</p>
<center><img src="/content/images/2017/09/taiwan_trip_4.jpg" /></center>
<p>I slept almost all the way to Taipei, not that I wanted to. But it was almost forced by my body. Effects of not sleeping at all
the previous night I guess.</p>
<center><img src="/content/images/2017/09/taiwan_trip_5.jpg" /></center>
<p>Arrival at Taipei airport was at around the afternoon, hunger caved in</p>
<center><img src="/content/images/2017/09/taiwan_trip_7.jpg" /></center>
<p>Luckily there was a Subway near the exit terminal and it was quite the lunch. Now came the time to actually travel back to my hotel room.</p>
<p>I should remind you that very few people actually can speak English back at Taiwan and if you are not comfortable asking
questions to strangers without a guide with, you are pretty much doomed.</p>
<center><img src="/content/images/2017/09/taiwan_trip_6.jpg" /></center>
<p>I was staying over at the northern part of Taiwan and it’s extremely pretty. You have a very good mix of traditional restaurants mixed with the new age restaurants and all the swanky stuff. Especially, I found quite a lot of national banks littered around the place I was staying.</p>
<center><img src="/content/images/2017/09/taiwan_trip_8.jpg" /></center>
<p>If you are starting out from the airport towards Nangang, you would be having some choices on how one would be willing to travel. Was suggested by people to take a cab, but the what the heck right? Took a metro where I had to change two stations with absolutely no English written over the directions. I dunno how did I not get lost.</p>
<p>And boy was my room not clean!</p>
<center><img src="/content/images/2017/09/taiwan_trip_9.jpg" /></center>
<p>The fluffy bed, the cosy cushions, soft carpet. It was amazing. Even the bathroom, a good nice bath tub with some good marble. Trust me, this is the most sophisticated commode that I have ever come across. Literally had to use Google translate for this.</p>
<center><img src="/content/images/2017/09/taiwan_trip_10.jpg" /></center>
<p>I had this itch of roaming around a little bit, still a sleepyhead. But couldn’t help it. This city is so beautiful! I had the most amazing noodles back in a local Taiwanese resto. It’s very different to eat in the sense that it’s quite thick than the normal noodles that people have here back in India. And you get a pretty generous helping of it too.</p>
<center><img src="/content/images/2017/09/taiwan_trip_12.jpg" /></center>
<p>On the way back to our hotel room, stopped by the local store. It was quite intriguing that you could see almost everything at the local store. Some localites were having their regular sip of coffee and chilling around, found a guy studying in the corner too. Possibly from a university.</p>
<center><img src="/content/images/2017/09/taiwan_trip_13.jpg" /></center>
<p>Came back, crashed on my bed, played some good music and before I knew it, I was fast asleep.</p>
<p>The morning after, I could really feel the pain in my shoulders carrying two bags over the city and trust me. It’s not a pleasant feeling.</p>
<p>Roamed a little bit more before leaving for the venue where the conference was gonna happen.</p>
<center><img src="/content/images/2017/09/taiwan_trip_11.jpg" /></center>
<p>The venue was a research institution which would be surrounded by woods. The campus was huge and was filled with lush greenery. It was a pretty place.</p>
<p>Here’s a small vlog from my first day back there.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/CuSWjAaysiM" frameborder="0" allowfullscreen=""></iframe>
<h2 id="pycon-taiwan">PyCon Taiwan</h2>
<p>The next morning was to be the first day of PyCon Taiwan, where I was slotted to give a talk later on the third day of the conference. I was excited as well as scared as this was gonna be my first international conference. Add that it was gonna be my first PyCon Talk.</p>
<p>As with the last 4 PyCon India’s that I have attended, this was quite different in the sense that it had a lot more diversity in terms of the demographics where people were coming from. We had people coming from China, South Korea, Japan, Australia and the rest of Asia Pacific. It was amazing to see how PyCon brought all of us together.</p>
<center><img src="/content/images/2017/09/taiwan_trip_14.jpg" /></center>
<p>After the talks ended, we all headed to Marriott, where we were to have the Speaker Night. I swear, I didn’t have such good sea food anywhere else. Sorry Goa, but that’s the truth.</p>
<center><img src="/content/images/2017/09/taiwan_trip_15.jpg" /></center>
<p>Had a lot of good conversations about open source, work culture back in Taiwan and Asia at large and how it impacted one’s health, work life balance and what not. We all made promises of staying in touch. And yes, some of us still are.</p>
<center><img src="/content/images/2017/09/taiwan_trip_16.jpg" /></center>
<p>I crashed over my bed and that was the end of the tiresome day for me.</p>
<p>Morning was a breeze, had some great sleep after dozing off talking back home and some friends. Breakfast equally good from yesterday’s dinner.</p>
<center><img src="/content/images/2017/09/taiwan_trip_17.jpg" /></center>
<p>Full with this good of a breakfast, left for the 2nd day of the conference. All the talks which I attended were top notch and the speakers really put in a lot of effort to prepare the talks.</p>
<p>Over the alleys, outside the talk halls. I struck some great conversations.</p>
<p>After the talks for the day ended, and some great dinner. We are all in for a treat to a great performance by my good friend Adrian. Trust me, he has his way with music. Crazy show!</p>
<center><img src="/content/images/2017/09/taiwan_trip_18.jpg" /></center>
<p>The show ended with a we decided to visit Taipei 101, formerly the world’s tallest building.</p>
<p>We got a cab and off we went. Taipei was more than a acquaintance now, I would remember that cars would actually stop when there was a green signal at the zebra crossing, the rules were actually being followed.</p>
<p>The streets were bustling like ever before, we could see a huge influx of the local residents as well as a huge amount of tourists. Streets performers, speed painters and what not. The one which drew a lot of attention was a blind man playing violin with his dog watchfully sitting with him.</p>
<p>Xinyin District is one of the well off parts of the city. Lot of big name shops and brands. You name it you have it.</p>
<center><img src="/content/images/2017/09/taiwan_trip_19.jpg" /></center>
<p>Most of the levels on the ground floor of Taipei 101 are filled with big name shops and eateries. Some of the levels above are filled with offices. A famous one being that of Google’s. Too bad I couldn’t go up and see the balancing weights which were made to counter earthquakes. Real piece of engineering.</p>
<center><img src="/content/images/2017/09/taiwan_trip_20.jpg" /></center>
<p>On the way back, we were passing through the City Square and there seemed to be some festival of a kind going on there, or maybe a carnival. Well I really don’t know what to call that. There were skimpily dressed girls dancing over the stages which had poles and there were men clicking photos with them and in one case one dude literally joining them on the stage. Not posting the pictures for those reasons.</p>
<p>And to my surprise, we met Andrew Godwin and Russell Keith-Magee, they shared the same surprised look seeing the carnival. And off we walked to the nearest metro station which would take us back to our Hotel. Both Andrew and Russell were kind enough to indulge in a great conversation during this time. Russell talked about funding open source projects to sustaining oneself while continuing contributions towards them.</p>
<p>Andrew and I had a long chat about his motivations towards Django channels and what excites him about open source. I was amazed by the humbleness of both.</p>
<center><img src="/content/images/2017/09/taiwan_trip_21.jpg" /></center>
<p>We settled for the night at our respective rooms and I slept like a log. Needed some good sleep as I had my talk slotted for tomorrow. I was nervous, scared and excited all at the same time.</p>
<p>The day finally came and I think I did okayish while giving my talk, will not say that I didn’t have any rough edges while giving the talk but I guess I have a lot of space for improvement.</p>
<center><img src="/content/images/2017/09/taiwan_trip_32.jpg" /></center>
<iframe width="560" height="315" src="https://www.youtube.com/embed/_Yif7EEOCy8" frameborder="0" allowfullscreen=""></iframe>
<p>The day ended with me leaving the venue after all the talks ended, next stop being the famous night markets of Taiwan. Had heard and read a lot about them. I ended up at the Raohe Street Night Market which is one of the oldest night markets in Songshan District.</p>
<p>Took a metro till Songshan Station and the place was as lively as it had been described in the other places.</p>
<p>You would find all kinds of cuisine there. It was a non-vegetarians delight to say the least.</p>
<p>Squids, octopus, dried fish and fruits, numerous kinds of noodles and what not.</p>
<center><img src="/content/images/2017/09/taiwan_trip_23.jpg" /></center>
<center><img src="/content/images/2017/09/taiwan_trip_24.jpg" /></center>
<center><img src="/content/images/2017/09/taiwan_trip_25.jpg" /></center>
<p>Next up was Lungshan Temple of Manka in Wanhua District. The temple was built in Taipei in 1738 by settlers from Fujian during Qing rule in honor of Guanyin. This temple has served as a municipal, guild and self-defence centre, as well as a house of worship.</p>
<p>You could feel the calmness setting in your mind by listening to the slow chants by the people which they were reciting in order to complete their prayers to their deity.</p>
<p>I just sat there for some time at a corner just to relax a while, to forget the bustling noise of the city and all the fatigue I had.</p>
<center><img src="/content/images/2017/09/taiwan_trip_22.jpg" /></center>
<center><img src="/content/images/2017/09/taiwan_trip_26.jpg" /></center>
<center><img src="/content/images/2017/09/taiwan_trip_27.jpg" /></center>
<p>There was another night market just a little up the temple, this night market was filled with massage parlours and some really old restaurants which were literally serving snake soup and the like. Nope, I didn’t have any of that. Was not feeling excited.</p>
<center><img src="/content/images/2017/09/taiwan_trip_28.jpg" /></center>
<center><img src="/content/images/2017/09/taiwan_trip_29.jpg" /></center>
<p>On the way back, while passing through the Lunshan metro station, which was known for fortune readers. You would find one shop at every other corner in the street. It was crazy, I was literally being hounded by people asking me to come in. This is some serious business out there for some people. Got some souvenirs on the way back to the airport from this kind lady who was very inquisitive about what I was doing here back in Taipie. When she leart that I had come to give a talk at a tech conference, she was very surprised that I was here all alone while being so young. Not sure how one can react to that. Anyways, had a good small chat about both our cultures and it was very interesting to know her perspective.</p>
<center><img src="/content/images/2017/09/taiwan_trip_30.jpg" /></center>
<p>My flight was set for departure early in the morning for Bangkok, had got nothing but a lot of time to kill at the airport as I reached at around 10pm.</p>
<p>While charging my phone, I met this girl who was from Macau pursuing her bachelors in Computer Science at Taiwan. It was interesting to get to know that back at her place, they have a real scarcity for good universities and that most people have to come out of there to get an education. She just came back from home after a short vacation. Just as we were discussing about the night markets of Taiwan, a guy joined us in the conversation and added his view point about how this is what actually built the culture of Taiwan, how it provided food and moneny on the table of the average Tiwanese citizen who had opened up a shop at one of the night markets.</p>
<p>Fast forward a few hours, I had the pleasure of talking to his wife who was from Philippines and his little baby boy. He talked about his life here back at Taiwan about how he had to run away from South Afica because he lost his job as a teacher back in the early 1990’s. He talked at length about how he left everything behind to come here and how he missed his family, how he could not meet his dying father who was suffering from cancer back at South Africa.</p>
<p>Those were the sad parts, he got really emotional talking about how his wife was facing problems about immigration back here in Taiwan and how bad the living conditions were back in Philippines.</p>
<p>Matter of fact, he had his flight for Philippines slotted for the next morning as he was going to meet his wife and son. I was very happy that he was able to reconcile with his bad past and look forward to the future of his family.</p>
<center><img src="/content/images/2017/09/taiwan_trip_31.jpg" /></center>
<p>Night ended and we bade good bye as I left to catch my flight to Bangkok. While I was sitting and latching my seat belt, I remembered the last few days here in Taiwan. It was a great journey indeed, and I am very grateful to the people back in Taiwan to be very accepting and kind.</p>
<p>Until next time Taiwan :)</p>
GSoC 2017 with oVirt - Ending Notes
2017-09-03T00:00:00+00:00
https://www.tasdikrahman.com/2017/09/03/Ending-notes-GSoC-2017-oVirt
<h2 id="start-of-coding-period">Start of Coding Period</h2>
<blockquote>
<p>This was also the time when I started my first contributions to oVirt. I surely remember getting just started with Ansible which was to be used in the project with oVirt.</p>
</blockquote>
<p>Lukas was very patient to help me with understanding the parts which would be used by me in the project and was very patient in every step.</p>
<p>Right from the application period of March, 2017. Every week has been a new learning experience for me.</p>
<p>I had a lot of side projects but never had I worked with an organisation this big and many developers behind it. It was an experience it in it’s own.</p>
<p>During this phase, I was travelling in between during the phase 2 due to relocation to a different state. That was from Bangalore to Dehradun. Quite a pretty place if you ask me.</p>
<p>Would sometimes work out of the local coffee shop and trust me internet can be a problem sometimes if you cross your FUP.</p>
<blockquote>
<p>There were times when I literally got badly stuck in a bug or a problem which I couldn’t get my head through which also taught me that if you stick to the problems long enough they can be solved</p>
</blockquote>
<p>This was also the time when I got into a habit of tracking everything related to side projects or GSoC.</p>
<center><img src="/content/images/2017/09/gsoc_final_post_1.jpeg" /></center>
<p>The above is back from my 1st evaluation.</p>
<center><img src="/content/images/2017/09/gsoc_final_post_2.jpeg" /></center>
<p>Continuing the tradition for the last month</p>
<p>Not the cleanest, but works for me.</p>
<p>I was lucky to be given a chance to speak at PyCon Taiwan 2017.</p>
<center><img src="/content/images/2017/09/gsoc_final_post_3.jpeg" /></center>
<p>(From left: Subhankar, founders of function Space and me on the extreme right)</p>
<p>From attending my first conference in 2014 to giving a talk at one of the PyCon’s. It sure has been a journey.</p>
<center><img src="/content/images/2017/09/gsoc_final_post_4.jpeg" /></center>
<h2 id="ending-notes">Ending notes</h2>
<p>As the coding period ends, I would like to thank my mentor Lukáš Svatý, for being there for my silly questions and giving me the direction, without whom nothing would have been possible and also the community members and people in the oVirt project. Without their guidance, I would be lost.</p>
<p>Here is an assimilated link to all my contributions and blog posts so far.</p>
<script src="https://gist.github.com/9931d1ad3b59ff350d4b1b08340be3a0.js"> </script>
<h2 id="takeaways-for-me">Takeaways for me</h2>
<ul>
<li>Experience of working remotely with a group of extremely talented people</li>
<li>Code formatting. Readability is important</li>
<li>Writing documentation which others can understand.</li>
<li>Reusing preexisting code for my own needs (looking at you engine-remote-db playbook)</li>
<li>Identifying refactoring needs. Did some extensive rewrites to the remote DWH PR</li>
<li>Writing code with performance in mind.</li>
</ul>
<h2 id="future-work">Future Work</h2>
<p>Continuing the work for Metrics Playbook deployment</p>
<p>As I write the last post for phase 3. I look back at the things contributing to oVirt has taught, things which will be there with me even after this. And I am glad that I submitted that proposal that day :)</p>
<p>Until next time!</p>
Second Phase - GSoC, work on 3 VM setup of oVirt installation
2017-07-30T00:00:00+00:00
https://www.tasdikrahman.com/2017/07/30/Second-Phase-GSoC-2017-oVirt
<p>It has been a week since the Phase 2 results are out. And we proceed to the last and final Phase of the GSoC. I couldn’t blog regularly in the last phase due to many reasons, which I want to change this Phase.
But anyhow, this was collectively the output of my work in Phase 2</p>
<h3 id="setup">Setup</h3>
<p>This approach will follow a 3 box VM setup.</p>
<p>For clarity sake, the VM’s can be assumed for now as</p>
<ul>
<li>VM A: <code class="language-plaintext highlighter-rouge">engine.ovirt.org</code>: which stored <code class="language-plaintext highlighter-rouge">engine</code> db and the main engine installation.</li>
<li>VM B: <code class="language-plaintext highlighter-rouge">dwhservice.ovirt.org</code>: which will host the dwh service</li>
<li>VM C: <code class="language-plaintext highlighter-rouge">dwhdb.ovirt.org</code>: which will store the <code class="language-plaintext highlighter-rouge">ovirt_engine_history</code> db</li>
</ul>
<h5 id="on-vm-a">On VM A</h5>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">engine</span>
<span class="na">vars</span><span class="pi">:</span>
<span class="na">ovirt_engine_type</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ovirt-engine'</span>
<span class="na">ovirt_engine_version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">4.1'</span>
<span class="na">ovirt_engine_organization</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dwhmanualenginetest.ovirt.org'</span>
<span class="na">ovirt_engine_admin_password</span><span class="pi">:</span> <span class="s1">'</span><span class="s">secret'</span>
<span class="na">ovirt_rpm_repo</span><span class="pi">:</span> <span class="s1">'</span><span class="s">http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'</span>
<span class="na">ovirt_engine_organization</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ovirt.org'</span>
<span class="na">ovirt_engine_dwh_db_host</span><span class="pi">:</span> <span class="s1">'</span><span class="s">remotedwh.ovirt.org'</span>
<span class="na">ovirt_engine_dwh_db_configure</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">roles</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-common</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-install-packages</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-setup</span>
</code></pre></div></div>
<p>Running this would be</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook site.yml <span class="nt">-i</span> inventory <span class="nt">--skip-tags</span> skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook site.yml <span class="nt">-i</span> inventory <span class="nt">--skip-tags</span> skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
<span class="o">[</span>WARNING]: While constructing a mapping from /Users/tasdikrahman/development/gsoc/ovirt-ansible/site.yml, line 4, column 5,
found a duplicate dict key <span class="o">(</span>ovirt_engine_organization<span class="o">)</span><span class="nb">.</span> Using last defined value only.
PLAY <span class="o">[</span>engine] <span class="k">*****************************************************************************************************************</span>
TASK <span class="o">[</span>Gathering Facts] <span class="k">********************************************************************************************************</span>
ok: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-common : complain <span class="k">if </span>no ovirt <span class="nb">source </span>is specified] <span class="k">****************************************************************</span>
skipping: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-common : <span class="nb">install </span>libselinux-python <span class="k">for </span>ansible] <span class="k">*******************************************************************</span>
ok: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-common : creating directory repo-backup <span class="k">in </span>yum.repos.d] <span class="k">***********************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-common : create repository backup] <span class="k">********************************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-common : copy repository files] <span class="k">***********************************************************************************</span>
TASK <span class="o">[</span>ovirt-common : <span class="nb">install </span>rpm repository package] <span class="k">**************************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-common : create repository files] <span class="k">*********************************************************************************</span>
TASK <span class="o">[</span>ovirt-engine-install-packages : yum <span class="nb">install </span>engine] <span class="k">*********************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : check <span class="k">if </span>ovirt-engine running <span class="o">(</span>health page<span class="o">)]</span> <span class="k">*******************************************************</span>
FAILED - RETRYING: check <span class="k">if </span>ovirt-engine running <span class="o">(</span>health page<span class="o">)</span> <span class="o">(</span>2 retries left<span class="o">)</span><span class="nb">.</span>
FAILED - RETRYING: check <span class="k">if </span>ovirt-engine running <span class="o">(</span>health page<span class="o">)</span> <span class="o">(</span>1 retries left<span class="o">)</span><span class="nb">.</span>
fatal: <span class="o">[</span>engine.ovirt.org]: FAILED! <span class="o">=></span> <span class="o">{</span><span class="s2">"attempts"</span>: 2, <span class="s2">"changed"</span>: <span class="nb">false</span>, <span class="s2">"content"</span>: <span class="s2">""</span>, <span class="s2">"failed"</span>: <span class="nb">true</span>, <span class="s2">"msg"</span>: <span class="s2">"Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>"</span>, <span class="s2">"redirected"</span>: <span class="nb">false</span>, <span class="s2">"status"</span>: <span class="nt">-1</span>, <span class="s2">"url"</span>: <span class="s2">"http://engine.ovirt.org/ovirt-engine/services/health"</span><span class="o">}</span>
...ignoring
TASK <span class="o">[</span>ovirt-engine-setup : copy default answerfile] <span class="k">***************************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : copy custom answer file] <span class="k">***************************************************************************</span>
skipping: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : update setup packages] <span class="k">*****************************************************************************</span>
skipping: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : run engine-setup with answerfile] <span class="k">******************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : check state of database] <span class="k">***************************************************************************</span>
ok: <span class="o">[</span>engine.ovirt.org]
▽
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
TASK <span class="o">[</span>ovirt-engine-setup : check state of engine] <span class="k">*****************************************************************************</span>
ok: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : restart of ovirt-engine service] <span class="k">*******************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : Open port 5432 <span class="k">for </span>opening connection to remote DWH <span class="k">if </span>not being setup on this host] <span class="k">***************</span>
changed: <span class="o">[</span>engine.ovirt.org] <span class="o">=></span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>5432<span class="o">)</span>
TASK <span class="o">[</span>ovirt-engine-setup : check health status of page] <span class="k">***********************************************************************</span>
FAILED - RETRYING: check health status of page <span class="o">(</span>12 retries left<span class="o">)</span><span class="nb">.</span>
FAILED - RETRYING: check health status of page <span class="o">(</span>11 retries left<span class="o">)</span><span class="nb">.</span>
FAILED - RETRYING: check health status of page <span class="o">(</span>10 retries left<span class="o">)</span><span class="nb">.</span>
FAILED - RETRYING: check health status of page <span class="o">(</span>9 retries left<span class="o">)</span><span class="nb">.</span>
ok: <span class="o">[</span>engine.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-setup : clean tmp files] <span class="k">***********************************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
RUNNING HANDLER <span class="o">[</span>ovirt-engine-setup : Reload firewalld] <span class="k">***********************************************************************</span>
changed: <span class="o">[</span>engine.ovirt.org]
PLAY RECAP <span class="k">********************************************************************************************************************</span>
engine.ovirt.org : <span class="nv">ok</span><span class="o">=</span>16 <span class="nv">changed</span><span class="o">=</span>10 <span class="nv">unreachable</span><span class="o">=</span>0 <span class="nv">failed</span><span class="o">=</span>0
</code></pre></div></div>
<h5 id="on-vm-c">On VM C</h5>
<p>On <code class="language-plaintext highlighter-rouge">testdwhdb.ovirt.org</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ssh root@dwhdb.ovirt.org
<span class="o">[</span>root@dwhdb ~]# <span class="nb">hostname
</span>dwhdb.ovirt.org
<span class="o">[</span>root@dwhdb ~]# yum <span class="nb">install </span>postgresql-server <span class="nt">-y</span> <span class="o">></span> /dev/null
<span class="o">[</span>root@dwhdb ~]# su <span class="nt">-l</span> postgres <span class="nt">-c</span> <span class="s2">"/usr/bin/initdb --locale=en_US.UTF8 --auth='ident' --pgdata=/var/lib/pgsql/data/"</span>
The files belonging to this database system will be owned by user <span class="s2">"postgres"</span><span class="nb">.</span>
This user must also own the server process.
The database cluster will be initialized with locale <span class="s2">"en_US.UTF8"</span><span class="nb">.</span>
The default database encoding has accordingly been <span class="nb">set </span>to <span class="s2">"UTF8"</span><span class="nb">.</span>
The default text search configuration will be <span class="nb">set </span>to <span class="s2">"english"</span><span class="nb">.</span>
fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database <span class="k">in</span> /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects<span class="s1">' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
Success. You can now start the database server using:
/usr/bin/postgres -D /var/lib/pgsql/data
or
/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
[root@dwhdb ~]#
[root@dwhdb ~]# systemctl start postgresql.service
[root@dwhdb ~]# systemctl enable postgresql.service
Created symlink from /etc/systemd/system/multi-user.target.wants/postgresql.service to /usr/lib/systemd/system/postgresql.service.
[root@dwhdb ~]# su - postgres
-bash-4.2$ psql
psql (9.2.18)
Type "help" for help.
postgres=# create role ovirt_engine_history with login encrypted password '</span>password<span class="s1">';
CREATE ROLE
postgres=# create database ovirt_engine_history owner ovirt_engine_history template template0 encoding '</span>UTF8<span class="s1">' lc_collate '</span>en_US.UTF-8<span class="s1">' lc_ctype '</span>en_US.UTF-8<span class="s1">';
postgres=# \q
-bash-4.2$ exit
[root@dwhdb ~]# tail -3 /var/lib/pgsql/data/pg_hba.conf
host ovirt_engine_history ovirt_engine_history xxx.xx.3.0/24 md5 # engine
host ovirt_engine_history ovirt_engine_history xxx.xx.11.0/24 md5 # dwhservice
[root@dwhdb ~]#
[root@dwhdb ~]# sed -n '</span>59,66p<span class="s1">' /var/lib/pgsql/data/postgresql.conf
listen_addresses = '</span><span class="k">*</span><span class="s1">' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to '</span>localhost<span class="s1">'; use '</span><span class="k">*</span><span class="s1">' for all
# (change requires restart)
#port = 5432 # (change requires restart)
# Note: In RHEL/Fedora installations, you can'</span>t <span class="nb">set </span>the port number here<span class="p">;</span>
<span class="c"># adjust it in the service file instead.</span>
max_connections <span class="o">=</span> 150 <span class="c"># (change requires restart)</span>
<span class="o">[</span>root@dwhdb ~]# systemctl start firewalld
<span class="o">[</span>root@dwhdb ~]# firewall-cmd <span class="nt">--zone</span><span class="o">=</span>public <span class="nt">--add-port</span><span class="o">=</span>5432/tcp <span class="nt">--permanent</span>
success
<span class="o">[</span>root@dwhdb ~]# firewall-cmd <span class="nt">--reload</span>
success
<span class="o">[</span>root@dwhdb ~]# iptables <span class="nt">-S</span> | <span class="nb">grep </span>5432
<span class="nt">-A</span> IN_public_allow <span class="nt">-p</span> tcp <span class="nt">-m</span> tcp <span class="nt">--dport</span> 5432 <span class="nt">-m</span> conntrack <span class="nt">--ctstate</span> NEW <span class="nt">-j</span> ACCEPT
<span class="o">[</span>root@dwhdb ~]#
</code></pre></div></div>
<p>So now that we have both the <code class="language-plaintext highlighter-rouge">engine</code> and <code class="language-plaintext highlighter-rouge">ovirt_engine_history</code> in place. We need to set up the VM which will setup up the dwh service on it.</p>
<h4 id="dwh-service-vm">dwh service VM</h4>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ssh root@dwhservice.ovirt.org
<span class="o">[</span>root@dwhservice ~]# <span class="nb">hostname
</span>dwhservice.ovirt.org
<span class="o">[</span>root@dwhservice ~]# yum <span class="nb">install</span> <span class="nt">-y</span> http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm <span class="o">></span> /dev/null
<span class="o">[</span>root@dwhservice ~]# yum <span class="nb">install</span> <span class="nt">-y</span> ovirt-engine-dwh-setup <span class="o">></span> /dev/null
http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/7/x86_64/repodata/fe30bd7c1f6f8d6f4007e9096a0c0fdd305c550f8c136ba86793577edc9dc571-updateinfo.xml.bz2: <span class="o">[</span>Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article
https://access.redhat.com/articles/1320623
If above article doesn<span class="s1">'t help to resolve this issue please create a bug on https://bugs.centos.org/
warning: /var/cache/yum/x86_64/7/ovirt-4.1-epel/packages/libtommath-0.42.0-5.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
warning: /var/cache/yum/x86_64/7/ovirt-4.1/packages/ovirt-engine-lib-4.1.4.2-1.el7.centos.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID fe590cb7: NOKEY
Importing GPG key 0xFE590CB7:
Userid : "oVirt <infra@ovirt.org>"
Fingerprint: 31a5 d783 7fad 7cb2 86cd 3469 ab8c 4f9d fe59 0cb7
Package : ovirt-release41-4.1.4-1.el7.centos.noarch (installed)
From : /etc/pki/rpm-gpg/RPM-GPG-ovirt-4.1
Importing GPG key 0x352C64E5:
Userid : "Fedora EPEL (7) <epel@fedoraproject.org>"
Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
[root@dwhservice ~]# engine-setup
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: ['</span>/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf<span class="s1">']
Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
Version: otopi-1.6.2 (otopi-1.6.2-1.el7.centos)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
--== PRODUCT OPTIONS ==--
Configure Data Warehouse on this host (Yes, No) [Yes]:
--== PACKAGES ==--
[ INFO ] Checking for product updates...
[ INFO ] No product updates found
--== NETWORK CONFIGURATION ==--
Host fully qualified DNS name of this server [dwhservice.ovirt.org]:
[WARNING] Failed to resolve dwhservice.ovirt.org using DNS, it can be resolved only locally
Setup can automatically configure the firewall on this system.
Note: automatic configuration of the firewall may overwrite current settings.
Do you want Setup to configure the firewall? (Yes, No) [Yes]:
The following firewall managers were detected on this system: firewalld
Firewall manager to configure (firewalld): firewalld
[ INFO ] firewalld will be configured as firewall manager.
Host fully qualified DNS name of the engine server []: engine.ovirt.org
Setup will need to do some actions on the remote engine server. Either automatically, using ssh as root to access it, or you will be prompted to manually perform each such action.
Please choose one of the following:
1 - Access remote engine server using ssh as root
2 - Perform each action manually, use files to copy content around
(1, 2) [1]:
ssh port on remote engine server [22]:
root password on remote engine server engine.ovirt.org:
--== DATABASE CONFIGURATION ==--
Where is the DWH database located? (Local, Remote) [Local]: Remote
ATTENTION
Manual action required.
Please create database for ovirt-engine use. Use the following commands as an example:
create role ovirt_engine_history with login encrypted password '</span><password><span class="s1">';
create database ovirt_engine_history owner ovirt_engine_history
template template0
encoding '</span>UTF8<span class="s1">' lc_collate '</span>en_US.UTF-8<span class="s1">'
lc_ctype '</span>en_US.UTF-8<span class="s1">';
Make sure that database can be accessed remotely.
DWH database host [localhost]: dwhdb.ovirt.org
DWH database port [5432]:
DWH database secured connection (Yes, No) [No]:
DWH database name [ovirt_engine_history]:
DWH database user [ovirt_engine_history]:
DWH database password:
Please provide the following credentials for the Engine database.
They should be found on the Engine server in '</span>/etc/ovirt-engine/engine.conf.d/10-setup-database.conf<span class="s1">'.
Engine database host []: engine.ovirt.org
Engine database port [5432]:
Engine database secured connection (Yes, No) [No]:
Engine database name [engine]:
Engine database user [engine]:
Engine database password:
--== OVIRT ENGINE CONFIGURATION ==--
--== STORAGE CONFIGURATION ==--
--== PKI CONFIGURATION ==--
--== APACHE CONFIGURATION ==--
--== SYSTEM CONFIGURATION ==--
--== MISC CONFIGURATION ==--
Please choose Data Warehouse sampling scale:
(1) Basic
(2) Full
(1, 2)[2]:
--== END OF CONFIGURATION ==--
[ INFO ] Stage: Setup validation
--== CONFIGURATION PREVIEW ==--
Firewall manager : firewalld
Update Firewall : True
Host FQDN : dwhservice.ovirt.org
Engine database secured connection : False
Engine database user name : engine
Engine database name : engine
Engine database host : engine.ovirt.org
Engine database port : 5432
Engine database host name validation : False
DWH installation : True
DWH database secured connection : False
DWH database host : dwhdb.ovirt.org
DWH database user name : ovirt_engine_history
DWH database name : ovirt_engine_history
DWH database port : 5432
DWH database host name validation : False
Please confirm installation settings (OK, Cancel) [OK]:
[ INFO ] Stage: Transaction setup
[ INFO ] Stopping dwh service
[ INFO ] Stage: Misc configuration
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Creating/refreshing DWH database schema
[ INFO ] Generating post install configuration file '</span>/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf<span class="s1">'
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
--== SUMMARY ==--
[ INFO ] Starting dwh service
Please restart the engine by running the following on engine.ovirt.org :
# service ovirt-engine restart
This is required for the dashboard to work.
--== END OF SUMMARY ==--
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
[ INFO ] Generating answer file '</span>/var/lib/ovirt-engine/setup/answers/20170802052510-setup.conf<span class="s1">'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of setup completed successfully
[root@dwhservice ~]# service restart ovirt-engine-dwh
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.
[root@dwhservice ~]#
[root@dwhservice ~]# service ovirt-engine-dwhd restart
Redirecting to /bin/systemctl restart ovirt-engine-dwhd.service
</span></code></pre></div></div>
<p>Now restart the service <code class="language-plaintext highlighter-rouge">systemctl restart ovirt-engine</code> on the <code class="language-plaintext highlighter-rouge">testengine.ovirt.org</code> VM and go to <a href="https://testengine.ovirt.org/ovirt-engine/">https://testengine.ovirt.org/ovirt-engine/</a></p>
<h3 id="automatinng-it-using-ansible">Automatinng it using Ansible</h3>
<p>The architecture of the overall system is similar to the one above.</p>
<p>Setup the <code class="language-plaintext highlighter-rouge">testengine.ovirt.org</code> the way we did the setup in the first step.</p>
<p>Next would be setup the postgresql instance which would host our <code class="language-plaintext highlighter-rouge">ovirt_engine_history</code>, we can do</p>
<h5 id="on-vm-c-1">On VM C</h5>
<p>the VM has to be setup for the <code class="language-plaintext highlighter-rouge">ovirt_engine_history</code> VM which is to be used by <code class="language-plaintext highlighter-rouge">dwhservice.ovirt.org</code> VM.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">dwhdb</span>
<span class="na">vars</span><span class="pi">:</span>
<span class="c1"># the below vars are explained in `install-postgresql/defaults/main.yml` and </span>
<span class="c1"># also the other configurable variables are placed there</span>
<span class="na">engine_vm_network_cidr</span><span class="pi">:</span> <span class="s1">'</span><span class="s">139.162.45.0/24'</span> <span class="c1"># Network where the Engine VM lies</span>
<span class="na">dwhservice_vm_network_cidr</span><span class="pi">:</span> <span class="s1">'</span><span class="s">139.162.61.0/24'</span>
<span class="na">ovirt_engine_dwh_db_password</span><span class="pi">:</span> <span class="s1">'</span><span class="s">password'</span>
<span class="na">roles</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-remote-dwh-setup/install-postgresql</span>
</code></pre></div></div>
<p>Running this would be</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook site.yml <span class="nt">-i</span> inventory
PLAY <span class="o">[</span>dwhdb] <span class="k">******************************************************************************************************************</span>
TASK <span class="o">[</span>Gathering Facts] <span class="k">********************************************************************************************************</span>
ok: <span class="o">[</span>testdwhdb.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] <span class="k">********************************************</span>
fatal: <span class="o">[</span>testdwhdb.ovirt.org]: FAILED! <span class="o">=></span> <span class="o">{</span><span class="s2">"changed"</span>: <span class="nb">false</span>, <span class="s2">"failed"</span>: <span class="nb">true</span>, <span class="s2">"msg"</span>: <span class="s2">"Could not find the requested service postgresql: host"</span><span class="o">}</span>
...ignoring
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : <span class="nb">install </span>postgresql] <span class="k">**************************************************</span>
changed: <span class="o">[</span>testdwhdb.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : Check <span class="k">if </span>the db is initialized] <span class="k">**************************************</span>
ok: <span class="o">[</span>testdwhdb.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : Initialize the postgresql db] <span class="k">****************************************</span>
<span class="o">[</span>WARNING]: Consider using <span class="s1">'become'</span>, <span class="s1">'become_method'</span>, and <span class="s1">'become_user'</span> rather than running su
changed: <span class="o">[</span>testdwhdb.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : Start postgresql.service] <span class="k">********************************************</span>
changed: <span class="o">[</span>testdwhdb.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : creating directory <span class="k">for </span>sql scripts <span class="k">in</span> /tmp/ansible-sql] <span class="k">**************</span>
changed: <span class="o">[</span>testdwhdb.ovirt.org]
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : copy SQL scripts] <span class="k">****************************************************</span>
changed: <span class="o">[</span>testdwhdb.ovirt.org] <span class="o">=></span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>ovirt-engine-dwh-db-user-create.sql<span class="o">)</span>
changed: <span class="o">[</span>testdwhdb.ovirt.org] <span class="o">=></span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>ovirt-engine-dwh-db-create.sql<span class="o">)</span>
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : create engine DWH DB and user] <span class="k">***************************************</span>
changed: <span class="o">[</span>testdwhdb.ovirt.org] <span class="o">=></span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>ovirt-engine-dwh-db-user-create.sql<span class="o">)</span>
changed: <span class="o">[</span>testdwhdb.ovirt.org] <span class="o">=></span> <span class="o">(</span><span class="nv">item</span><span class="o">=</span>ovirt-engine-dwh-db-create.sql<span class="o">)</span>
TASK <span class="o">[</span>ovirt-engine-remote-dwh-setup/install-postgresql : Adding engine and dwhservice vm IP<span class="s1">'s in the dwhdb conf to be acessed remotely] ***
changed: [testdwhdb.ovirt.org] => (item=host ovirt_engine_history ovirt_engine_history 139.162.45.0/24 md5 # engine)
changed: [testdwhdb.ovirt.org] => (item=host ovirt_engine_history ovirt_engine_history 139.162.61.0/24 md5 # dwhservice)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Edit the config file /var/lib/pgsql/data/postgresql.conf] ************
changed: [testdwhdb.ovirt.org] => (item=sed -i -- '</span>s/max_connections <span class="o">=</span> 100/max_connections <span class="o">=</span> 150/g<span class="s1">' /var/lib/pgsql/data/postgresql.conf)
changed: [testdwhdb.ovirt.org] => (item=sed -i "60ilisten_addresses = '</span><span class="k">*</span><span class="s1">'" /var/lib/pgsql/data/postgresql.conf)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable firewalld and open up port 5432] ******************************
changed: [testdwhdb.ovirt.org] => (item=systemctl start firewalld)
changed: [testdwhdb.ovirt.org] => (item=firewall-cmd --zone=public --add-port=5432/tcp --permanent)
changed: [testdwhdb.ovirt.org] => (item=firewall-cmd --reload)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Restart postgresql for the loading the newer configs] ****************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] ********************************************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : clean tmp files] *****************************************************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable postgresql.service to start at boot time] *********************
ok: [testdwhdb.ovirt.org]
PLAY RECAP ********************************************************************************************************************
testdwhdb.ovirt.org : ok=16 changed=12 unreachable=0 failed=0
</span></code></pre></div></div>
<h5 id="on-vm-b">On VM B</h5>
<p>Now VM which hosts the <code class="language-plaintext highlighter-rouge">dwhservice</code> needs to be setup.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">dwhservice</span>
<span class="na">vars</span><span class="pi">:</span>
<span class="na">ovirt_engine_type</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ovirt-engine'</span>
<span class="na">ovirt_engine_version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">4.1'</span>
<span class="na">ovirt_rpm_repo</span><span class="pi">:</span> <span class="s1">'</span><span class="s">http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'</span>
<span class="na">ovirt_engine_host_root_passwd</span><span class="pi">:</span> <span class="s1">'</span><span class="s">pycon2017'</span> <span class="c1"># the root password of the host where ovirt-engine is installed</span>
<span class="na">ovirt_engine_firewall_manager</span><span class="pi">:</span> <span class="s1">'</span><span class="s">firewalld'</span>
<span class="na">ovirt_engine_host_fqdn</span><span class="pi">:</span> <span class="s1">'</span><span class="s">testengine.ovirt.org'</span> <span class="c1"># FQDN of the ovirt-engine installation host, should be resolvable from the new DWH host</span>
<span class="na">ovirt_engine_db_host</span><span class="pi">:</span> <span class="s1">'</span><span class="s">testengine.ovirt.org'</span>
<span class="na">ovirt_engine_dwh_db_host</span><span class="pi">:</span> <span class="s1">'</span><span class="s">testdwhdb.ovirt.org'</span>
<span class="na">ovirt_engine_dwh_db_password</span><span class="pi">:</span> <span class="s1">'</span><span class="s">password'</span>
<span class="na">ovirt_engine_history_db_on_dwhservice_host</span><span class="pi">:</span> <span class="s">False</span>
<span class="na">roles</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-common</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-install-packages</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-remote-dwh-setup</span>
</code></pre></div></div>
<p>Running this would be</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook site.yml <span class="nt">-i</span> inventory <span class="nt">--skip-tags</span> skip_yum_install_ovirt_engine,skip_yum_install_ovirt_engine_dwh <span class="nt">-vvv</span>
</code></pre></div></div>
<p>Now restart <code class="language-plaintext highlighter-rouge">ovirt-engine</code> in VMA by doing a <code class="language-plaintext highlighter-rouge">systemctl restart ovirt-engine</code></p>
<center><img src="/content/images/2017/07/successfull_deployment_3_vm.png" /></center>
<h3 id="issues-faced">Issues Faced</h3>
<p>The very first thing which I faced as an issue was that, there was no proper documentation in the oVirt Downstream docs in place (as on now) in place. This made me research more about the things. Good in one way, anyway I raised report on bugzilla about here. https://bugzilla.redhat.com/show_bug.cgi?id=1475706
There was another snag like the previous one where an OTOPI variable was not being logged in the answerfiles which led me to</p>
<h3 id="reference-commits">Reference Commits</h3>
<ul>
<li><a href="https://github.com/rhevm-qe-automation/ovirt-ansible/pull/137/commits/d5152c388d2de3bdc4d955561d4a190a9187b62">Adds remote postgresql installation playbook</a></li>
<li><a href="https://github.com/rhevm-qe-automation/ovirt-ansible/pull/137/commits/8dd39bdd46e1a73e9ccf8d09bf7e96de2f3d38cc">Adds role for the dwh service setup along with the remaining three VMS’s</a></li>
</ul>
<h3 id="future-work">Future Work</h3>
<ul>
<li>Cleanup of the PR’s opened so far.</li>
<li>Work on test upgrade playbook.</li>
<li>role for ovirt metrics deployment</li>
<li>role for direct upgrade </li>
</ul>
<h3 id="references">References</h3>
<ul>
<li>http://www.ovirt.org/documentation/install-guide/appe-Preparing_a_Remote_PostgreSQL_Database_for_Use_with_the_oVirt_Engine/</li>
<li>https://www.linode.com/docs/databases/postgresql/how-to-install-postgresql-relational-databases-on-centos-7</li>
<li>https://www.postgresql.org/docs/8.1/static/sql-createrole.html</li>
<li>https://support.rackspace.com/how-to/postgresql-creating-and-dropping-roles/</li>
</ul>
<p>I have to say, I got stuck a lot while trying this out and there have numerous times where I was just about to give up. But my mentor was always there to support my questions and doubts. Thanks for that Lukas :)</p>
<p>More to come</p>
Week 3 and 4, GSoC 2017 - dozens of cloud vm's, ansibling, finding bugs, testing
2017-06-28T00:00:00+00:00
https://www.tasdikrahman.com/2017/06/28/Week-3-and-4-delving-deeper-GSoC-2017-oVirt
<p>At last I have got a hold of IRC’s and I declare my love for <a href="https://irssi.org/">irssi</a>. The combination of <a href="https://tmux.github.io/">tmux</a> and irssi is a boon for me. I tried different clients like <a href="http://www.weechat.org/">weechat</a>,<a href="http://www.epicsol.org//">Epic</a> but I found irssi to be more appealing, quite frankly use any of these two if you are looking for something using which you can chat on irc’s on the terminal.</p>
<p>My current installation of irssi is there on my VPC hosted on linode. For logging my chat’s and conversations, I have setup logrotate daemon to log the channels and private chats in their respective directories.</p>
<p>So what does it look like? Definitely a huge shift from the web chat interface and more customisable.</p>
<center><img src="/content/images/2017/06/irssi_tmux.png" /></center>
<p>I love the current setup. Just one thing which I think is missing out from here is that I would like to get a notification when I get a message or someone mentions my name in a channel. Will have to look that up.</p>
<p>You can try <a href="limechat.net/mac/">Limechat</a> if you are on a MAC.</p>
<p>Between, I will be there with the handle <code class="language-plaintext highlighter-rouge">tasdikrahman</code> hanging around the channels #ovirt on OFTC as well as on freenode, mostly #ansible and #gsoc.</p>
<p>Anyways, that’s for my recent addition of irc love.</p>
<p>Talking about work, I have been working on my existing PR looking into the feedback which my mentor and other members of the infra team have given.</p>
<p>And right now, I am working on adding playbooks for configuring a remote DWH on an ovirt-engine setup.</p>
<h2 id="prelude">Prelude</h2>
<p>Reading up the docs, the DWH is a historic database that allows users to create reports over a static API using business intelligence suites that enable you to monitor the system. It contains the ETL (Extract Transform Load) process created using Talend Open Studio and DB scripts to create a working history DB.</p>
<p>This history database(<code class="language-plaintext highlighter-rouge">ovirt_engine_history</code> to be precise) can be utilized by any application to extract a range of information at the data center, cluster, and host levels.</p>
<p>Simply put, it’s a BI tool for your ovirt installation.</p>
<center><img src="/content/images/2017/06/ovirt-arch.png" /></center>
<p>oVirt Engine uses PostgreSQL 8.4.x as the database platform to store information about the state of the virtualization environment, its configuration and performance. At install time, ovirt engine creates a PostgreSQL database called <code class="language-plaintext highlighter-rouge">engine</code>. You have the option to either install this on the same host or a different host(remote)</p>
<p>The <code class="language-plaintext highlighter-rouge">ovirt-engine-dwh</code> package creates a second database called <code class="language-plaintext highlighter-rouge">ovirt_engine_history</code>, which contains historical configuration information and statistical metrics collected every minute over time from the engine operational database. Tracking the changes to the database provides information on the objects in the database, enabling the user to analyze activity, enhance performance, and resolve difficulties.</p>
<p>You can track <a href="http://www.ovirt.org/documentation/data-warehouse/Tracking_configuration_history/">configuration data</a>, <a href="http://www.ovirt.org/documentation/data-warehouse/Recording_statistical_history/">statistical data</a> into your <code class="language-plaintext highlighter-rouge">ovirt_engine_history</code> database based on your needs.</p>
<h2 id="but-why-should-someone-run-the-dwh-service-on-a-different-host">But why should someone run the DWH service on a different host?</h2>
<p>Quite obviously running a whole lot of services on one host would be very memory heavy. And what happens when someone tries to squeeze out a lot of things from a single instance?</p>
<p>There is a decrease in performance output due to fight amongst the processes for resources. So it’s very logical to distribute your services to different hosts if possible.</p>
<center><img src="/content/images/2017/06/ovirt-two-box-arch.jpg" /></center>
<p>The above architecture basically shows the simple setup of the ovirt-engine and dwh all being on the same host.</p>
<p>Now even when you are trying to seperate out the load by distributing the services running in a host, you further have a flexibily offered by oVirt here.</p>
<p>Some being.</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">engine</code> db itself being on a differnt host rather than on the host where ovirt-engine is running.</li>
<li>DWH being installed on a different host than the one where ovirt-engine is installed.(feature being available only after oVirt Engine 3.5)</li>
<li>DWH being installed on a different host and the DWH db being on a 3rd host.</li>
</ol>
<p>The first one is covered by the ansible role <a href="https://github.com/rhevm-qe-automation/ovirt-ansible/tree/master/roles/ovirt-engine-remote-db">ovirt-engine-remote-db</a> which lets you do so</p>
<p>I tried tackling the 2nd one on the list in my 3rd week.</p>
<center><img src="/content/images/2017/06/ovirt-one-box-arch.jpg" /></center>
<p>Had the initial discussions about <a href="https://github.com/rhevm-qe-automation/ovirt-ansible/issues/9">issue #9</a> with Lukas and we narrowed down to the end goals and tasks for that particular issue on github.</p>
<p>The docs are pretty clear on the process itself and a bit of poring over them made the things which were to be done.</p>
<p>As I had mentioned in my <a href="https://medium.com/@tasdikrahman/week-1-and-2-gsoc-2017-travel-code-good-food-1f91d34344b9">earlier post</a> about not automating something which you haven’t achieved manually yet.</p>
<p>That applied here too.</p>
<p>For starters, I rebuilt 2x2gig Centos7 boxes on Linode which were hardened using my own custom ansible playbook called <a href="https://github.com/tasdikrahman/ansible-bootstrap-server">ansible-bootstrap-server</a>. I had a good lesson about security when <a href="https://edgecoders.com/learnings-from-analyzing-my-compromised-server-linode-cd3be62dc286">one of my servers got compromised</a> so this one was a must.</p>
<h2 id="some-assumptions-for-the-two-systems">Some assumptions for the two systems</h2>
<center><img src="/content/images/2017/06/vm-assumptions.jpg" /></center>
<p>Expanding on the first one, it should be able to do so by using the FQDN’s of the respective VM’s in question.</p>
<p>To add to the above,</p>
<ul>
<li>Ports 80, 443, 5432(<code class="language-plaintext highlighter-rouge">postgres</code>) should be open on both these hosts.</li>
</ul>
<p>You can open the above ports using <code class="language-plaintext highlighter-rouge">firewalld</code> using (assuming you are <code class="language-plaintext highlighter-rouge">root</code>)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>firewall-cmd <span class="nt">--zone</span><span class="o">=</span>public <span class="nt">--add-port</span><span class="o">=</span>80/tcp <span class="nt">--permanent</span>
<span class="nv">$ </span><span class="c"># further adding any rules to appended to iptables</span>
<span class="nv">$ </span>firewalld-cmd <span class="nt">--reload</span> <span class="c"># for the rules to take effect</span>
<span class="nv">$ </span>iptables <span class="nt">-S</span> <span class="c"># to check whether the changes have taken effect</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">iptables</code> would spit every rule defined out there for your system. That would get ugly real quick. A cleaner way would be to do a</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@dwhtest-3-engine ~]# iptables-save | <span class="nb">grep </span>443
<span class="nt">-A</span> IN_public_allow <span class="nt">-p</span> tcp <span class="nt">-m</span> tcp <span class="nt">--dport</span> 443 <span class="nt">-m</span> conntrack <span class="nt">--ctstate</span> NEW <span class="nt">-j</span> ACCEPT
<span class="o">[</span>root@dwhtest-3-engine ~]#
</code></pre></div></div>
<p>After these, the machine is just set for configuration.</p>
<h2 id="manual-installation">Manual installation</h2>
<p>Install and setup ovirt-engine on machine A, ovirt-engine-dwh on machine B, see that dwhd on B collects data from the engine on A.</p>
<p>On A:</p>
<h4 id="installing-dwh-on-a-remote-host-from-the-one-having-ovirt-engine">Installing DWH on a remote host from the one having ovirt-engine</h4>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@dwhtest-3-engine ~]# <span class="nb">hostname
</span>dwhtest-3-engine.ovirt.org
<span class="o">[</span>root@dwhtest-3-engine ~]# yum <span class="nb">install </span>http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm
<span class="o">[</span>root@dwhtest-3-engine ~]# yum <span class="nt">-y</span> <span class="nb">install </span>ovirt-engine
<span class="o">[</span>root@dwhtest-3-engine ~]# engine-setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Initializing
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment setup
Configuration files: <span class="o">[</span><span class="s1">'/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf'</span>, <span class="s1">'/etc/ovirt-engine-setup.conf.d/10-packaging.conf'</span><span class="o">]</span>
Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170625105958-cyvseu.log
Version: otopi-1.6.2 <span class="o">(</span>otopi-1.6.2-1.el7.centos<span class="o">)</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment packages setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Programs detection
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment customization
<span class="nt">--</span><span class="o">==</span> PRODUCT OPTIONS <span class="o">==</span><span class="nt">--</span>
Configure Engine on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Configure Image I/O Proxy on this host? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Configure WebSocket Proxy on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Please note: Data Warehouse is required <span class="k">for </span>the engine. If you choose to not configure it on this host, you have to configure it on a remote host, and <span class="k">then </span>configure the engine on this host so that it can access the database of the remote Data Warehouse host.
Configure Data Warehouse on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]: No
Configure VM Console Proxy on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
<span class="nt">--</span><span class="o">==</span> PACKAGES <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Checking <span class="k">for </span>product updates...
<span class="o">[</span> INFO <span class="o">]</span> No product updates found
<span class="nt">--</span><span class="o">==</span> NETWORK CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Host fully qualified DNS name of this server <span class="o">[</span>dwhtest-3-engine.ovirt.org]:
<span class="o">[</span>WARNING] Failed to resolve dwhtest-3-engine.ovirt.org using DNS, it can be resolved only locally
Setup can automatically configure the firewall on this system.
Note: automatic configuration of the firewall may overwrite current settings.
Do you want Setup to configure the firewall? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
The following firewall managers were detected on this system: firewalld
Firewall manager to configure <span class="o">(</span>firewalld<span class="o">)</span>: firewalld
<span class="o">[</span> INFO <span class="o">]</span> firewalld will be configured as firewall manager.
<span class="nt">--</span><span class="o">==</span> DATABASE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Where is the Engine database located? <span class="o">(</span>Local, Remote<span class="o">)</span> <span class="o">[</span>Local]:
Setup can configure the <span class="nb">local </span>postgresql server automatically <span class="k">for </span>the engine to run. This may conflict with existing applications.
Would you like Setup to automatically configure postgresql and create Engine database, or prefer to perform that manually? <span class="o">(</span>Automatic, Manual<span class="o">)</span> <span class="o">[</span>Automatic]:
<span class="nt">--</span><span class="o">==</span> OVIRT ENGINE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Engine admin password:
Confirm engine admin password:
<span class="o">[</span>WARNING] Password is weak: it is based on a dictionary word
Use weak password? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]: Yes
Application mode <span class="o">(</span>Virt, Gluster, Both<span class="o">)</span> <span class="o">[</span>Both]:
<span class="nt">--</span><span class="o">==</span> STORAGE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Default SAN wipe after delete <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]:
<span class="nt">--</span><span class="o">==</span> PKI CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Organization name <span class="k">for </span>certificate <span class="o">[</span>ovirt.org]:
<span class="nt">--</span><span class="o">==</span> APACHE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Setup can configure the default page of the web server to present the application home page. This may conflict with existing applications.
Do you wish to <span class="nb">set </span>the application as the default page of the web server? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Setup can configure apache to use SSL using a certificate issued from the internal CA.
Do you wish Setup to configure that, or prefer to perform that manually? <span class="o">(</span>Automatic, Manual<span class="o">)</span> <span class="o">[</span>Automatic]:
<span class="nt">--</span><span class="o">==</span> SYSTEM CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Configure an NFS share on this server to be used as an ISO Domain? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]:
<span class="nt">--</span><span class="o">==</span> MISC CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="nt">--</span><span class="o">==</span> END OF CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Setup validation
<span class="o">[</span>WARNING] Cannot validate host name settings, reason: resolved host does not match any of the <span class="nb">local </span>addresses
<span class="o">[</span>WARNING] Warning: Not enough memory is available on the host. Minimum requirement is 4096MB, and 16384MB is recommended.
Do you want Setup to <span class="k">continue</span>, with amount of memory less than recommended? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]: Yes
<span class="nt">--</span><span class="o">==</span> CONFIGURATION PREVIEW <span class="o">==</span><span class="nt">--</span>
Application mode : both
Default SAN wipe after delete : False
Firewall manager : firewalld
Update Firewall : True
Host FQDN : dwhtest-3-engine.ovirt.org
Configure <span class="nb">local </span>Engine database : True
Set application as default page : True
Configure Apache SSL : True
Engine database secured connection : False
Engine database user name : engine
Engine database name : engine
Engine database host : localhost
Engine database port : 5432
Engine database host name validation : False
Engine installation : True
PKI organization : ovirt.org
DWH installation : False
Configure <span class="nb">local </span>DWH database : False
Configure Image I/O Proxy : True
Configure VMConsole Proxy : True
Configure WebSocket Proxy : True
Please confirm installation settings <span class="o">(</span>OK, Cancel<span class="o">)</span> <span class="o">[</span>OK]:
<span class="o">[</span> INFO <span class="o">]</span> Stage: Transaction setup
<span class="o">[</span> INFO <span class="o">]</span> Stopping engine service
<span class="o">[</span> INFO <span class="o">]</span> Stopping ovirt-fence-kdump-listener service
<span class="o">[</span> INFO <span class="o">]</span> Stopping Image I/O Proxy service
<span class="o">[</span> INFO <span class="o">]</span> Stopping vmconsole-proxy service
<span class="o">[</span> INFO <span class="o">]</span> Stopping websocket-proxy service
<span class="o">[</span> INFO <span class="o">]</span> Stage: Misc configuration
<span class="o">[</span> INFO <span class="o">]</span> Stage: Package installation
<span class="o">[</span> INFO <span class="o">]</span> Stage: Misc configuration
<span class="o">[</span> INFO <span class="o">]</span> Upgrading CA
<span class="o">[</span> INFO <span class="o">]</span> Initializing PostgreSQL
<span class="o">[</span> INFO <span class="o">]</span> Creating PostgreSQL <span class="s1">'engine'</span> database
<span class="o">[</span> INFO <span class="o">]</span> Configuring PostgreSQL
<span class="o">[</span> INFO <span class="o">]</span> Creating CA
<span class="o">[</span> INFO <span class="o">]</span> Creating/refreshing Engine database schema
<span class="o">[</span> INFO <span class="o">]</span> Configuring Image I/O Proxy
<span class="o">[</span> INFO <span class="o">]</span> Setting up ovirt-vmconsole proxy helper PKI artifacts
<span class="o">[</span> INFO <span class="o">]</span> Setting up ovirt-vmconsole SSH PKI artifacts
<span class="o">[</span> INFO <span class="o">]</span> Configuring WebSocket Proxy
<span class="o">[</span> INFO <span class="o">]</span> Creating/refreshing Engine <span class="s1">'internal'</span> domain database schema
<span class="o">[</span> INFO <span class="o">]</span> Generating post <span class="nb">install </span>configuration file <span class="s1">'/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Transaction commit
<span class="o">[</span> INFO <span class="o">]</span> Stage: Closing up
<span class="o">[</span> INFO <span class="o">]</span> Starting engine service
<span class="o">[</span> INFO <span class="o">]</span> Restarting ovirt-vmconsole proxy service
<span class="nt">--</span><span class="o">==</span> SUMMARY <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Restarting httpd
Please use the user <span class="s1">'admin@internal'</span> and password specified <span class="k">in </span>order to login
The engine requires access to the Data Warehouse database.
Data Warehouse was not <span class="nb">set </span>up. Please <span class="nb">set </span>it up on some other machine and configure access to it on the engine.
Web access is enabled at:
http://dwhtest-3-engine.ovirt.org:80/ovirt-engine
https://dwhtest-3-engine.ovirt.org:443/ovirt-engine
Internal CA 1F:E4:07:AF:E2:63:27:8F:4A:E2:A1:8D:F2:63:9B:BA:29:F7:3D:21
SSH fingerprint: 38:ca:86:6d:82:50:ca:c7:9c:03:ed:bc:0b:3a:e5:33
<span class="o">[</span>WARNING] Warning: Not enough memory is available on the host. Minimum requirement is 4096MB, and 16384MB is recommended.
<span class="nt">--</span><span class="o">==</span> END OF SUMMARY <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170625105958-cyvseu.log
<span class="o">[</span> INFO <span class="o">]</span> Generating answer file <span class="s1">'/var/lib/ovirt-engine/setup/answers/20170625112418-setup.conf'</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Pre-termination
<span class="o">[</span> INFO <span class="o">]</span> Stage: Termination
<span class="o">[</span> INFO <span class="o">]</span> Execution of setup completed successfully
<span class="o">[</span>root@dwhtest-3-engine ~]#
</code></pre></div></div>
<p>So ovirt-engine is set on machine A.</p>
<p>The only difference here being when running <code class="language-plaintext highlighter-rouge">engine-setup</code> on the host, answering No to configuring Data Warehouse:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
Configure Data Warehouse on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]: No
...
</code></pre></div></div>
<p>The other thing to note in the above installation is the <code class="language-plaintext highlighter-rouge">answerfile</code> which was generated out of the installation process.</p>
<h2 id="answerfile">Answerfile?</h2>
<p>Tools like <code class="language-plaintext highlighter-rouge">engine-setup</code>, <code class="language-plaintext highlighter-rouge">engine-cleanup</code> and <code class="language-plaintext highlighter-rouge">ovirt-engine-rename</code> which are based out on OTOPI(key value pair store of data and configuration), all generate something called as an answerfile upon completion of their (be it successful or erroneous).</p>
<p>This file is placed under <code class="language-plaintext highlighter-rouge">/var/lib/ovirt-engine/setup/answers/</code> ending with a <code class="language-plaintext highlighter-rouge">*.conf</code>.</p>
<center><img src="/content/images/2017/06/state0-to-state1.jpg" /></center>
<p>So have system A which is present on some state0, “S0” in this sense includes basically everything relevant - versions of relevant packages, enabled repos, history (such as previous runs of these tools, other data accumulated over time, etc), other manual configuration, etc.</p>
<p>The expected way to use these answer files is:</p>
<ol>
<li>Have a system A in some state S0</li>
<li>Run one of the tools interactively, answer its questions as needed, let it create an answer file Ans1. Basically answering all the places where the program was waiting for <code class="language-plaintext highlighter-rouge">STDIN</code>.</li>
<li>System A is now in state S1.</li>
<li>Have some other system B in state S0, that you want to bring to state S1.</li>
<li>Run there the same tool with <code class="language-plaintext highlighter-rouge">–config-append=Ans1</code></li>
</ol>
<blockquote>
<p>When used this way, the tools should run unattended. If they still ask questions, it’s generally considered a bug.</p>
</blockquote>
<p>Which allows and makes way for automation.</p>
<p>Manually editing such an answer file is generally not supported/expected and should not be needed. You might do that to achieve special non-standard goals. If you do that, you should thoroughly verify that it works for you, and use in a controlled environment - same known initial state, same versions of relevant stuff, etc.</p>
<h2 id="leveraging-the-things-seen-in-this-answerfile">Leveraging the things seen in this answerfile</h2>
<p>We already have in place an ansible role for installing ovirt-engine along with DWH on the same host here named <a href="https://github.com/rhevm-qe-automation/ovirt-ansible/tree/master/roles/ovirt-engine-setup">ovirt-engine-setup</a>.</p>
<p>Having a look at an existing answerfile, take <code class="language-plaintext highlighter-rouge">ovirt-engine-setup/templates/answerfile_4.0_basic.txt.j2</code> for example.</p>
<p>As it’s based out on OTOPI, and I had for reference the newly generated anwerfile from the succesfull install on machine A.</p>
<p>What I really wanted was a <code class="language-plaintext highlighter-rouge">diff</code> between these two answerfiles, one which was generated when DWH was installed along with the engine on the same host and the other being when the DWH would be configured on a remote host</p>
<div class="language-patch highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">46c46
</span><span class="gd">< OVESETUP_DWH_CORE/enable=bool:True
</span><span class="p">---
</span><span class="gi">> OVESETUP_DWH_CORE/enable=bool:False
</span><span class="p">49c49
</span><span class="gd">< OVESETUP_DWH_DB/secured=bool:False
</span><span class="p">---
</span><span class="gi">> OVESETUP_DWH_DB/secured=none:None
</span><span class="p">52,53c52,54
</span><span class="gd">< OVESETUP_DWH_DB/host=str:localhost
< OVESETUP_DWH_DB/user=str:ovirt_engine_history
</span><span class="p">---
</span><span class="gi">> OVESETUP_DWH_DB/host=none:None
> OVESETUP_DWH_DB/user=none:None
> OVESETUP_DWH_DB/password=none:None
</span><span class="p">55c56
</span><span class="gd">< OVESETUP_DWH_DB/database=str:ovirt_engine_history
</span><span class="p">---
</span><span class="gi">> OVESETUP_DWH_DB/database=none:None
</span><span class="p">57c58
</span><span class="gd">< OVESETUP_DWH_DB/port=int:5432
</span><span class="p">---
</span><span class="gi">> OVESETUP_DWH_DB/port=none:None
</span><span class="p">60,61c61,62
</span><span class="gd">< OVESETUP_DWH_DB/securedHostValidation=bool:False
< OVESETUP_DWH_PROVISIONING/postgresProvisioningEnabled=bool:True
</span><span class="p">---
</span><span class="gi">> OVESETUP_DWH_DB/securedHostValidation=none:None
> OVESETUP_DWH_PROVISIONING/postgresProvisioningEnabled=bool:False
</span></code></pre></div></div>
<p>This gave me a clear idea of what I needed to do with the existing answer file to make it suitable for configuring a remote DWH.</p>
<div class="language-patch highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh">diff --git a/roles/ovirt-engine-setup/templates/answerfile_4.0_basic.txt.j2 b/roles/ovirt-engine-setup/templates/answerfile_4.0_basic.txt.j2
index 728a307..d60d03f 100644
</span><span class="gd">--- a/roles/ovirt-engine-setup/templates/answerfile_4.0_basic.txt.j2
</span><span class="gi">+++ b/roles/ovirt-engine-setup/templates/answerfile_4.0_basic.txt.j2
</span><span class="p">@@ -29,16 +29,29 @@</span> OVESETUP_DB/port=int:{ {ovirt_engine_db_port} }
OVESETUP_DB/filter=none:None
OVESETUP_DB/restoreJobs=int:2
OVESETUP_DB/securedHostValidation=bool:False
<span class="gi">+
+OVESETUP_DWH_DB/secured=none:None
+OVESETUP_DWH_DB/host=none:None
+OVESETUP_DWH_DB/user=none:None
+OVESETUP_DWH_DB/password=none:None
+OVESETUP_DWH_DB/database=none:None
+OVESETUP_DWH_DB/port=none:None
+
+OVESETUP_DWH_DB/dumper=str:pg_custom
</span> OVESETUP_DWH_DB/filter=none:None
OVESETUP_DWH_DB/restoreJobs=int:2
<span class="gi">+
+OVESETUP_DWH_DB/securedHostValidation=none:None
+
</span> OVESETUP_ENGINE_CORE/enable=bool:True
OVESETUP_CORE/engineStop=none:None
OVESETUP_SYSTEM/memCheckEnabled=bool:False
<span class="p">@@ -65,13 +78,21 @@</span> OVESETUP_CONFIG/imageioProxyConfig=bool:True
OVESETUP_PROVISIONING/postgresProvisioningEnabled=bool:True
OVESETUP_APACHE/configureRootRedirection=bool:True
OVESETUP_APACHE/configureSsl=bool:True
<span class="gi">+
+OVESETUP_DWH_CORE/enable=bool:False
+
</span> OVESETUP_DWH_CONFIG/scale=str:1
OVESETUP_DWH_CONFIG/dwhDbBackupDir=str:/var/lib/ovirt-engine-dwh/backups
OVESETUP_DWH_DB/restoreBackupLate=bool:True
OVESETUP_DWH_DB/disconnectExistingDwh=none:None
OVESETUP_DWH_DB/performBackup=none:None
<span class="gi">+
+OVESETUP_DWH_PROVISIONING/postgresProvisioningEnabled=bool:False
+
</span> OVESETUP_DB/password=str:{ {ovirt_engine_db_password} }
OVESETUP_RHEVM_DIALOG/confirmUpgrade=bool:True
OVESETUP_VMCONSOLE_PROXY_CONFIG/vmconsoleProxyConfig=bool:True
</code></pre></div></div>
<p>the variable <code class="language-plaintext highlighter-rouge">ovirt_engine_dwh_db_configure</code> would be a <code class="language-plaintext highlighter-rouge">bool</code> defined inside our play yaml file telling <code class="language-plaintext highlighter-rouge">ovirt-engine-setup</code> to not configure DWH on that host.</p>
<p>This modification sets us up for the first part.</p>
<h2 id="configuring-the-remote-dwh-manually-first">Configuring the remote DWH manually first</h2>
<p>If you are trying to access the web adming on machine A, it would show you an error like this.</p>
<center><img src="/content/images/2017/06/admin-panel-error-without-dwh.png" /></center>
<p>On machine B</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@dwhtest-3-dwh ~]# yum <span class="nt">-y</span> update <span class="o">></span> /dev/null
<span class="o">[</span>root@dwhtest-3-dwh ~]# yum <span class="nt">-y</span> <span class="nb">install </span>ovirt-engine-dwh-setup <span class="o">></span> /dev/null
<span class="o">[</span>root@dwhtest-3-dwh ~]# engine-setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Initializing
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment setup
Configuration files: <span class="o">[</span><span class="s1">'/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf'</span><span class="o">]</span>
Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170625114213-l4cku9.log
Version: otopi-1.6.2 <span class="o">(</span>otopi-1.6.2-1.el7.centos<span class="o">)</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment packages setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Programs detection
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment customization
<span class="nt">--</span><span class="o">==</span> PRODUCT OPTIONS <span class="o">==</span><span class="nt">--</span>
Configure Data Warehouse on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
<span class="nt">--</span><span class="o">==</span> PACKAGES <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Checking <span class="k">for </span>product updates...
<span class="o">[</span> INFO <span class="o">]</span> No product updates found
<span class="nt">--</span><span class="o">==</span> NETWORK CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Host fully qualified DNS name of this server <span class="o">[</span>dwhtest-3-dwh.ovirt.org]:
<span class="o">[</span>WARNING] Failed to resolve dwhtest-3-dwh.ovirt.org using DNS, it can be resolved only locally
Setup can automatically configure the firewall on this system.
Note: automatic configuration of the firewall may overwrite current settings.
Do you want Setup to configure the firewall? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
The following firewall managers were detected on this system: firewalld
Firewall manager to configure <span class="o">(</span>firewalld<span class="o">)</span>: firewalld
<span class="o">[</span> INFO <span class="o">]</span> firewalld will be configured as firewall manager.
Host fully qualified DNS name of the engine server <span class="o">[]</span>: dwhtest-3-engine.ovirt.org
Setup will need to <span class="k">do </span>some actions on the remote engine server. Either automatically, using ssh as root to access it, or you will be prompted to manually perform each such action.
Please choose one of the following:
1 - Access remote engine server using ssh as root
2 - Perform each action manually, use files to copy content around
<span class="o">(</span>1, 2<span class="o">)</span> <span class="o">[</span>1]:
ssh port on remote engine server <span class="o">[</span>22]:
root password on remote engine server dwhtest-3-engine.ovirt.org:
<span class="nt">--</span><span class="o">==</span> DATABASE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Where is the DWH database located? <span class="o">(</span>Local, Remote<span class="o">)</span> <span class="o">[</span>Local]:
Setup can configure the <span class="nb">local </span>postgresql server automatically <span class="k">for </span>the DWH to run. This may conflict with existing applications.
Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? <span class="o">(</span>Automatic, Manual<span class="o">)</span> <span class="o">[</span>Automatic]:
Please provide the following credentials <span class="k">for </span>the Engine database.
They should be found on the Engine server <span class="k">in</span> <span class="s1">'/etc/ovirt-engine/engine.conf.d/10-setup-database.conf'</span><span class="nb">.</span>
Engine database host <span class="o">[]</span>: dwhtest-3-engine.ovirt.org
Engine database port <span class="o">[</span>5432]:
Engine database secured connection <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]:
Engine database name <span class="o">[</span>engine]:
Engine database user <span class="o">[</span>engine]:
Engine database password:
<span class="nt">--</span><span class="o">==</span> OVIRT ENGINE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="nt">--</span><span class="o">==</span> STORAGE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="nt">--</span><span class="o">==</span> PKI CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="nt">--</span><span class="o">==</span> APACHE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="nt">--</span><span class="o">==</span> SYSTEM CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="nt">--</span><span class="o">==</span> MISC CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Please choose Data Warehouse sampling scale:
<span class="o">(</span>1<span class="o">)</span> Basic
<span class="o">(</span>2<span class="o">)</span> Full
<span class="o">(</span>1, 2<span class="o">)[</span>2]:
<span class="nt">--</span><span class="o">==</span> END OF CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Setup validation
<span class="nt">--</span><span class="o">==</span> CONFIGURATION PREVIEW <span class="o">==</span><span class="nt">--</span>
Firewall manager : firewalld
Update Firewall : True
Host FQDN : dwhtest-3-dwh.ovirt.org
Engine database secured connection : False
Engine database user name : engine
Engine database name : engine
Engine database host : dwhtest-3-engine.ovirt.org
Engine database port : 5432
Engine database host name validation : False
DWH installation : True
DWH database secured connection : False
DWH database host : localhost
DWH database user name : ovirt_engine_history
DWH database name : ovirt_engine_history
DWH database port : 5432
DWH database host name validation : False
Configure <span class="nb">local </span>DWH database : True
Please confirm installation settings <span class="o">(</span>OK, Cancel<span class="o">)</span> <span class="o">[</span>OK]:
<span class="o">[</span> INFO <span class="o">]</span> Stage: Transaction setup
<span class="o">[</span> INFO <span class="o">]</span> Stopping dwh service
<span class="o">[</span> INFO <span class="o">]</span> Stage: Misc configuration
<span class="o">[</span> INFO <span class="o">]</span> Stage: Package installation
<span class="o">[</span> INFO <span class="o">]</span> Stage: Misc configuration
<span class="o">[</span> INFO <span class="o">]</span> Initializing PostgreSQL
<span class="o">[</span> INFO <span class="o">]</span> Creating PostgreSQL <span class="s1">'ovirt_engine_history'</span> database
<span class="o">[</span> INFO <span class="o">]</span> Configuring PostgreSQL
<span class="o">[</span> INFO <span class="o">]</span> Creating/refreshing DWH database schema
<span class="o">[</span> INFO <span class="o">]</span> Generating post <span class="nb">install </span>configuration file <span class="s1">'/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Transaction commit
<span class="o">[</span> INFO <span class="o">]</span> Stage: Closing up
<span class="nt">--</span><span class="o">==</span> SUMMARY <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Starting dwh service
Please restart the engine by running the following on dwhtest-3-engine.ovirt.org :
<span class="c"># service ovirt-engine restart</span>
This is required <span class="k">for </span>the dashboard to work.
<span class="nt">--</span><span class="o">==</span> END OF SUMMARY <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170625114213-l4cku9.log
<span class="o">[</span> INFO <span class="o">]</span> Generating answer file <span class="s1">'/var/lib/ovirt-engine/setup/answers/20170625115725-setup.conf'</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Pre-termination
<span class="o">[</span> INFO <span class="o">]</span> Stage: Termination
<span class="o">[</span> INFO <span class="o">]</span> Execution of setup completed successfully
<span class="o">[</span>root@dwhtest-3-dwh ~]#
</code></pre></div></div>
<p>Restarting the ovirt-engine service on machine A starts the whole thing.</p>
<h3 id="debugging-tips">Debugging tips</h3>
<p>If for any reason, the admin panel is still giving you an error.</p>
<ul>
<li>Try checking the output of <code class="language-plaintext highlighter-rouge">iptables -S</code>, to see whether you are allowing incoming connections on port <code class="language-plaintext highlighter-rouge">5432</code> on both machines.</li>
<li>Are both hosts able to resolve to their public IP’s using the provided FQDN</li>
<li>Try entering those passwords slowly.</li>
<li>If all fails, the logs of each <code class="language-plaintext highlighter-rouge">engine-setup</code> command would be there for your rescue.</li>
</ul>
<h2 id="automating-it">Automating it</h2>
<center><img src="/content/images/2017/06/ansible-playbook-flow.jpg" /></center>
<p>For testing my newly created <code class="language-plaintext highlighter-rouge">ovirt-engine-install-remote-dwh</code>, provisioned 2x2gig CentOS 7 vms on linode</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasrahma at TASRAHMA-M-C2MT <span class="k">in</span> ~/development/gsoc/ovirt-ansible <span class="o">(</span>remote-dwh-fresh-engine-install●●<span class="o">)</span>
<span class="nv">$ </span>ssh root@dwhmanualenginetest.ovirt.org
root@dwhmanualenginetest.ovirt.org<span class="s1">'s password:
[root@dwhmanualenginetest ~]# hostname
dwhmanualenginetest.ovirt.org
[root@dwhmanualenginetest ~]# systemctl start firewalld
[root@dwhmanualenginetest ~]# firewall-cmd --zone=public --add-port=80/tcp --permanent
success
[root@dwhmanualenginetest ~]# firewall-cmd --zone=public --add-port=443/tcp --permanent
success
[root@dwhmanualenginetest ~]# firewall-cmd --zone=public --add-port=5432/tcp --permanent
success
[root@dwhmanualenginetest ~]# firewall-cmd --reload
success
[root@dwhmanualenginetest ~]# exit
logout
Connection to dwhmanualenginetest.ovirt.org closed.
tasrahma at TASRAHMA-M-C2MT in ~/development/gsoc/ovirt-ansible (remote-dwh-fresh-engine-install●●)
$ ssh root@dwhmanualdwhtest.ovirt.org
root@dwhmanualdwhtest.ovirt.org'</span>s password:
<span class="o">[</span>root@li1462-178 ~]# <span class="nb">hostname
</span>dwhmanualdwhtest.ovirt.org
<span class="o">[</span>root@li1462-178 ~]# vi /etc/hosts
<span class="o">[</span>root@li1462-178 ~]# systemctl start firewalld
<span class="o">[</span>root@li1462-178 ~]# firewall-cmd <span class="nt">--zone</span><span class="o">=</span>public <span class="nt">--add-port</span><span class="o">=</span>80/tcp <span class="nt">--permanent</span>
success
<span class="o">[</span>root@li1462-178 ~]# firewall-cmd <span class="nt">--zone</span><span class="o">=</span>public <span class="nt">--add-port</span><span class="o">=</span>443/tcp <span class="nt">--permanent</span>
success
<span class="o">[</span>root@li1462-178 ~]# firewall-cmd <span class="nt">--zone</span><span class="o">=</span>public <span class="nt">--add-port</span><span class="o">=</span>5432/tcp <span class="nt">--permanent</span>
success
<span class="o">[</span>root@li1462-178 ~]# firewall-cmd <span class="nt">--reload</span>
success
<span class="o">[</span>root@li1462-178 ~]# <span class="nb">exit
logout
</span>Connection to dwhmanualdwhtest.ovirt.org closed.
</code></pre></div></div>
<p>With these done, the servers are ready for my new ansible roles.</p>
<p>First would be the installation of the <code class="language-plaintext highlighter-rouge">ovirt-engine</code> host</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">engine</span>
<span class="na">vars</span><span class="pi">:</span>
<span class="na">ovirt_engine_type</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ovirt-engine'</span>
<span class="na">ovirt_engine_version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">4.1'</span>
<span class="na">ovirt_engine_organization</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dwhmanualenginetest.ovirt.org'</span>
<span class="na">ovirt_engine_admin_password</span><span class="pi">:</span> <span class="s1">'</span><span class="s">secret'</span>
<span class="na">ovirt_rpm_repo</span><span class="pi">:</span> <span class="s1">'</span><span class="s">http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'</span>
<span class="na">ovirt_engine_organization</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ovirt.org'</span>
<span class="na">ovirt_engine_dwh_db_host</span><span class="pi">:</span> <span class="s1">'</span><span class="s">remotedwh.ovirt.org'</span>
<span class="na">ovirt_engine_dwh_db_configure</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">roles</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-common</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-install-packages</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-setup</span>
</code></pre></div></div>
<p>Running this play file would be done like</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook site.yml <span class="nt">-i</span> inventory <span class="nt">--skip-tags</span> skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
</code></pre></div></div>
<p>After which you have to configure your remote DWH installation to the previous host which has the ovirt-engine installation which again has two possibilities</p>
<p>On your machine B which would be <code class="language-plaintext highlighter-rouge">dwhmanualdwhtest.ovirt.org</code> in this case</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span> <span class="s">dwh-remote</span>
<span class="na">vars</span><span class="pi">:</span>
<span class="na">ovirt_engine_type</span><span class="pi">:</span> <span class="s1">'</span><span class="s">ovirt-engine'</span>
<span class="na">ovirt_engine_version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">4.1'</span>
<span class="na">ovirt_rpm_repo</span><span class="pi">:</span> <span class="s1">'</span><span class="s">http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'</span>
<span class="na">ovirt_engine_host_root_passwd</span><span class="pi">:</span> <span class="s1">'</span><span class="s">admin123'</span> <span class="c1"># the root password of the host where ovirt-engine is installed</span>
<span class="na">ovirt_engine_firewall_manager</span><span class="pi">:</span> <span class="s1">'</span><span class="s">firewalld'</span>
<span class="na">ovirt_engine_db_host</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dwhmanualenginetest.ovirt.org'</span> <span class="c1"># FQDN of the ovirt-engine installation host, should be resolvable from the new DWH host</span>
<span class="na">ovirt_engine_host_fqdn</span><span class="pi">:</span> <span class="s1">'</span><span class="s">dwhmanualenginetest.ovirt.org'</span>
<span class="na">ovirt_engine_dwh_db_host</span><span class="pi">:</span> <span class="s1">'</span><span class="s">localhost'</span>
<span class="na">ovirt_dwh_on_dwh</span><span class="pi">:</span> <span class="s">True</span>
<span class="na">roles</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-common</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-install-packages</span>
<span class="pi">-</span> <span class="na">role</span><span class="pi">:</span> <span class="s">ovirt-engine-remote-dwh-setup</span>
</code></pre></div></div>
<p>Running this play file would be done like</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook site.yml <span class="nt">-i</span> inventory <span class="nt">--skip-tags</span> skip_yum_install_ovirt_engine,skip_yum_install_ovirt_engine_dwh
</code></pre></div></div>
<p>After this you have to restart your ovirt-engine on the host installed by doing a <code class="language-plaintext highlighter-rouge">service ovirt-engine restart</code></p>
<p>And you have the whole setup now.</p>
<h2 id="possibilty-of-the-database-of-dwh-on-3rd-machine">Possibilty of the database of dwh on 3rd machine</h2>
<p>I have not taken this case as of now in the new playbook which I wrote. I was able to figure out how to do it manually but the specific changes for getting this option to work using our roles is still pending.</p>
<h2 id="next-goals">Next goals</h2>
<ul>
<li>The very first thing to do now would be to modify the existing <code class="language-plaintext highlighter-rouge">ovirt-engine-setup-remote-dwh</code> that I wrote to include the case of putthing the DB of the DWH on a 3rd machine.</li>
<li>Also, Migration dwh from local machine of engine to remote machine in a new playbook which counts with already installed engine with local dwh and moved dwh to this remote server is also on the task list</li>
<li>Writing tests for the new additions</li>
</ul>
<h2 id="pitfalls">Pitfalls</h2>
<p>The most arcane error that I faced during this was when I was trying to automate the configuration of the remote DWH with the engine host.</p>
<center><img src="/content/images/2017/06/arcane-otopi-var-missing-ansible-error.png" /></center>
<p>From a first glance at the error traceback, it was clear that there was some error reading the engine hostname from the OTOPI based answerfile that I had generated.</p>
<p>I cross checked again and again from the answerfile which was generated from the manual installation of remote DWH as described in the process above, but to no avail.</p>
<p>Both of them looked just the same. I was not missing anything out from the original answerfile and they were basically the same thing minus the hostnames and passwords.</p>
<p>To be frank, I really didn’t know what was missing out here.</p>
<p>On a closer look at the manual input being given while installation and the answerfile being generated from it, I noticed that the engine HOST FQDN that we provide is not being logged in the OTOPI answerfile.</p>
<p>Cross checking with the answerfile being generated on a new host, and sure enough. There was a prompt for the host engine FQDN which confirmed my hunch.</p>
<center><img src="/content/images/2017/06/confirmed-hunch-prompt.png" /></center>
<p>This had to be the case! I mean there was no other explanation. I asked Lukas and he said this possibly might be a bug and suggested that I file a bug on bugzilla. Which I did here at <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1465859">https://bugzilla.redhat.com/show_bug.cgi?id=1465859</a> which lead me to my first bug report :)</p>
<p>Cross checking with the answerfile being generated on a new host, and sure enough. There was a prompt for the host engine FQDN which confirmed my hunch.</p>
<p>Anyway, now I just had to figure out a way of what was the OTOPI config being logged in the log files when I did enter this FQDN. Checking the logs for that particular engine-run, the logs showed me that the variable being logged.</p>
<p>It was <code class="language-plaintext highlighter-rouge">OVESETUP_ENGINE_CONFIG/fqdn</code>. I placed this on my template answerfile and added the remote engine host value to it.</p>
<p>And sure enough, it ran successfully :)</p>
<p>Crazy day! But that feeling when you successfully debug something. It’s always priceless no matter if it isn’t the first time you are doing so.</p>
<p>Until next time!</p>
<h2 id="links">Links</h2>
<ul>
<li><a href="https://github.com/rhevm-qe-automation/ovirt-ansible/compare/master...tasdikrahman:remote-dwh-fresh-engine-install">Github branch for the work being done on issue #9 namely <code class="language-plaintext highlighter-rouge">remote-dwh-fresh-engine-install</code></a></li>
<li><a href="https://github.com/rhevm-qe-automation/ovirt-ansible/issues/9">https://github.com/rhevm-qe-automation/ovirt-ansible/issues/9</a></li>
<li><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1465859">https://bugzilla.redhat.com/show_bug.cgi?id=1465859</a></li>
<li><a href="https://www.ovirt.org/documentation/architecture/architecture/">https://www.ovirt.org/documentation/architecture/architecture/</a></li>
<li><a href="http://www.ovirt.org/documentation/data-warehouse/Data_Warehouse_Guide/">http://www.ovirt.org/documentation/data-warehouse/Data_Warehouse_Guide/</a></li>
<li><a href="http://www.ovirt.org/documentation/data-warehouse/Migrating_Data_Warehouse_to_a_Separate_Machine/">http://www.ovirt.org/documentation/data-warehouse/Migrating_Data_Warehouse_to_a_Separate_Machine/</a></li>
<li><a href="http://www.ovirt.org/develop/release-management/features/engine/separate-dwh-host/">http://www.ovirt.org/develop/release-management/features/engine/separate-dwh-host/</a></li>
<li><a href="http://www.ovirt.org/develop/release-management/features/engine/migration-of-local-dwh-reports-to-remote/">http://www.ovirt.org/develop/release-management/features/engine/migration-of-local-dwh-reports-to-remote/</a></li>
<li><a href="https://www.ovirt.org/documentation/how-to/hosted-engine/">https://www.ovirt.org/documentation/how-to/hosted-engine/</a></li>
<li><a href="https://www.ovirt.org/develop/developer-guide/engine/engine-setup/">https://www.ovirt.org/develop/developer-guide/engine/engine-setup/</a></li>
</ul>
Week 1 and 2, GSoC 2017 - Travel, Code, Good food
2017-06-13T00:00:00+00:00
https://www.tasdikrahman.com/2017/06/13/Week-1-and-2-ovirt-engine-rename-GSoC-2017-oVirt-Redhat
<p>So it’s quite some time since I wrote a thing or two about the things which I have doing over the last 2 weeks.</p>
<p>This month has been a roller coaster ride if you ask me. Many reasons to it.</p>
<p>Some being that I travelled to <a href="http://www.tasdikrahman.com/2017/06/12/PyCon-Taiwan-2017-Taipei/">PyCon Taiwan</a> which marked my first international trip and also my first PyCon talk. Got my uni results. Fingers crossed but heck. I scored a perfect 10 in the last sem! The final version of <a href="https://github.com/tasdikrahman/trumporate/">trumporate’s</a> UI is almost done and me and Rituraj have to just put some final touches to (blogpost for the whole development process is pending. You can find the first <a href="http://www.tasdikrahman.com/2017/05/06/Making-of-trumporate-using-markovipy-generating-sentences-using-markov-chains-part-1/">one here</a>)</p>
<p>As for the GSoC work. I have been working closely with Lukas on the <code class="language-plaintext highlighter-rouge">engine-rename</code> role which has been assigned as one of the first tasks for the first review.</p>
<h2 id="approach">Approach</h2>
<blockquote>
<p>Don’t automate something which you haven’t achieved manually yet!</p>
</blockquote>
<p>I read this somewhere and the author was one of the employees of <a href="https://www.chef.io/chef/">chef software</a> at one of the chef conferences. Now don’t ask me what was I doing watching chef videos. I really don’t remember what led me to that. Thanks to youtube’s reco engines!</p>
<p>But never the less, I think the quote is quite true even in the literal sense.</p>
<p>I mean, if you are trying to repeat a task which consists of several small tasks with some automation tool. You ought to have gone through the manual process of doing that manually once. Especially when you are new to either the tool/process that you are trying to automate or new to the automation tool itself or both.</p>
<p>The reasoning behind this is very simple. As you are bound to get hiccups when trying to do the above, you will get frustrated trying to debug the errors that you are getting while automating the process.</p>
<p>I mean what would you be pinpointing your error to? The process which you are following? Is that wrong? Or the syntax or way-of-doing-things which your automation tool follows?</p>
<p>Hard to say if you ask me.</p>
<p>So with this in mind I first tried renaming an ovirt-engine installation task manually.</p>
<p>Nothing fancy, you find the exact steps over here from one of the ovirt <a href="https://www.ovirt.org/documentation/how-to/networking/changing-engine-hostname/">documentation pages</a></p>
<p>The command is very simple (assuming you are inside your ovirt-engine installation)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/usr/share/ovirt-engine/setup/bin/ovirt-engine-rename <span class="se">\</span>
<span class="nt">--newname</span><span class="o">=</span>ovirte1n.home.local <span class="se">\</span>
<span class="nt">--otopi-environment</span><span class="o">=</span><span class="s2">"OSETUP_RENAME/forceIgnoreAIAInCA=bool:'True' </span><span class="se">\</span><span class="s2">
OVESETUP_CORE/engineStop=bool:'True' </span><span class="se">\</span><span class="s2">
OSETUP_RENAME/confirmForceOverwrite=bool:'False'"</span>
</code></pre></div></div>
<p>The above tries, to the rename the engine-setup with the new name with minimal(read no) external interference.</p>
<p>More like <code class="language-plaintext highlighter-rouge">expect</code></p>
<p>so the value to the optional argument <code class="language-plaintext highlighter-rouge">--newname</code> would be the new engine-name that you wanna replace the older one with. This was given back at the time of the engine-installation but now you want it renamed.</p>
<p>When the <code class="language-plaintext highlighter-rouge">engine-setup</code> command is run in a clean environment, the command generates a number of certificates and keys that use the fully qualified domain name of the Manager supplied during the setup process. If the fully qualified domain name of the Manager must be changed later on (for example, due to migration of the machine hosting the Manager to a different domain), the records of the fully qualified domain name must be updated to reflect the new name. The <code class="language-plaintext highlighter-rouge">ovirt-engine-rename</code> command automates this task.</p>
<p>The <code class="language-plaintext highlighter-rouge">ovirt-engine-rename</code> command updates records of the fully qualified domain name of the Manager in the following locations:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
/etc/ovirt-engine/imageuploader.conf.d/10-engine-setup.conf
/etc/ovirt-engine/isouploader.conf.d/10-engine-setup.conf
/etc/ovirt-engine/logcollector.conf.d/10-engine-setup.conf
/etc/pki/ovirt-engine/cert.conf
/etc/pki/ovirt-engine/cert.template
/etc/pki/ovirt-engine/certs/apache.cer
/etc/pki/ovirt-engine/keys/apache.key.nopass
/etc/pki/ovirt-engine/keys/apache.p12
</code></pre></div></div>
<p>Once the command is executed (assuming you got no error’s), and with no surprises. The engine has been renamed. But there are some things yet to be done.</p>
<p>The next thing is to change the hostname. This is usually done by editing <code class="language-plaintext highlighter-rouge">/etc/hostname</code> and rebooting.</p>
<p>You can go with the <code class="language-plaintext highlighter-rouge">hostnamectl</code> command if you are on a RHEL based system. Even <code class="language-plaintext highlighter-rouge">hostname</code> command works if you. If you are using DNS to pull out the hostname, then prepare relevant DNS and/or <code class="language-plaintext highlighter-rouge">/etc/hosts</code> records for the new name.</p>
<p>Assuming you were accessing the web-portal before doing your engine-rename part. You will notice that you are note able to access the web interface</p>
<p>I was testing all this out on a 2gig Linode centos7 box (Love these guys). Right now it’s all manual provisioning. I have to take a look at <a href="https://www.terraform.io/">terraform</a> sometime later.</p>
<p>The current workflow for me would be to create a new VM and then provision it using the <a href="https://github.com/rhevm-qe-automation/ovirt-ansible/tree/master/roles/ovirt-engine-setup"><code class="language-plaintext highlighter-rouge">ovirt-engine-setup</code></a> role which set’s up the ovirt-engine for you. I have written a <a href="http://www.tasdikrahman.com/2017/05/24/Installing-ovirt-4.1-on-centos-7-using-ansible-linode-Google-Summer-of-Code-oVirt-2017/">blog post</a> here if you are curious about how to do so using an ansible role.</p>
<p>Once that is done, your path for the role to rename the engine is quite clear.</p>
<p>For my personal setup, I had to now change the entry in my local dev box <code class="language-plaintext highlighter-rouge">/etc/hosts</code> to the new hostname which I gave for the ovirt-engine.</p>
<p>Now open the new DNS mapping on your browser. And you should be able to see the WEB UI up.</p>
<h2 id="automating-it">Automating it</h2>
<p>Now as we have the roadmap to what needs to be done, I needed to write an ansible-role for it.</p>
<p>A very simple to guess edge case here is that the new engine name is same as the one already present. How do I check that?</p>
<p>Checking logs would be the answer for that. I needed to check under <code class="language-plaintext highlighter-rouge">/etc/ovirt-engine</code> for that. Namely the files below</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf
/etc/ovirt-engine/ovirt-vmconsole-proxy-helper.conf.d/10-setup.conf
/etc/ovirt-engine/isouploader.conf.d/10-engine-setup.conf
/etc/ovirt-engine/logcollector.conf.d/10-engine-setup.conf
</code></pre></div></div>
<p>The engine-name would be logged under these files.</p>
<p>What I wanted to have in place was check whether the new name is not the same as the existing name out there.</p>
<p>So if I did a <code class="language-plaintext highlighter-rouge">grep</code> recursively on the directory of <code class="language-plaintext highlighter-rouge">/etc/ovirt-engine</code> for the with the <code class="language-plaintext highlighter-rouge">new-name</code>, it would either return something (this case means the engine-name is already at the required state) or nothing at all(this case being when you the new-name is a new one and not the one already present)</p>
<p>Registering this grep value to a file on <code class="language-plaintext highlighter-rouge">/tmp/engine-rename-logs-current.grep</code> and then comparing it with a <code class="language-plaintext highlighter-rouge">template</code> file using <code class="language-plaintext highlighter-rouge">diff</code> which stores the expected grep result does the job.</p>
<p>If the diff with this expected file returns nothing, we are already at our required state.</p>
<p>If not the <code class="language-plaintext highlighter-rouge">engine-rename</code> command is run.</p>
<p>After which again a recursive grep is done again on <code class="language-plaintext highlighter-rouge">/etc/ovirt-engine</code> with the search parameter as the new engine name.</p>
<p>This result is registered and compared to the expected grep result. If this passes, we now have our ovirt-engine at the desired state. i.e the name of the engine was renamed successfully.</p>
<p>After which we are checking the engine health at the end of the role.</p>
<p>The handlers which were notified when creating the temporary files are executed by the playbook at this point which delete them from the file system.</p>
<h2 id="testing-all-the-above">Testing all the above</h2>
<center><img src="/content/images/2017/06/brace-yourselves-testing-is-coming.jpg" /></center>
<p>Now testing these changes out is the real thing.</p>
<p>We are using <code class="language-plaintext highlighter-rouge">docker</code> containers to test out the roles. I would say this is approach is quite fast and gives us the <code class="language-plaintext highlighter-rouge">works-everywhere</code> advantage.</p>
<p>We need two containers, one for the <code class="language-plaintext highlighter-rouge">engine</code> and one for the <code class="language-plaintext highlighter-rouge">remote-db</code> which we are using <a href="http://chrismeyersfsu.me/">Chris Meyer’s</a>, <a href="https://github.com/chrismeyersfsu/provision_docker/">provision_docker</a> tooling.</p>
<p>The first role in any test would be to provision the docker containers and make sure that they are reachable.</p>
<p><code class="language-plaintext highlighter-rouge">ovirt-ansible/tests/containers-deploy.yml</code> is the first role which is included as the first role to be executed in all of the <code class="language-plaintext highlighter-rouge">test-$.yml</code> files.</p>
<p>The above role inside, calls the role named <code class="language-plaintext highlighter-rouge">provision_docker</code> twice for provisioning the docker inventory groups specified as key:value pair to the role. Which would be nothing but the <code class="language-plaintext highlighter-rouge">engine</code> and the <code class="language-plaintext highlighter-rouge">remote-db</code> containers specified inside the <code class="language-plaintext highlighter-rouge">tests/inventory</code> file.</p>
<p>The other roles follow.</p>
<p>The role <code class="language-plaintext highlighter-rouge">ovirt-engine-rename</code> would be coming just before we would be running the role of <code class="language-plaintext highlighter-rouge">ovirt-engine-cleanup</code>.</p>
<p>Adding mine just above that will do the trick.</p>
<blockquote>
<p>The particular PR which covers the above <a href="https://github.com/rhevm-qe-automation/ovirt-ansible/pull/132/files">https://github.com/rhevm-qe-automation/ovirt-ansible/pull/132/files</a></p>
</blockquote>
<h2 id="few-gotchas">Few gotcha’s</h2>
<p>The command <code class="language-plaintext highlighter-rouge">$ hostnamectl set-hostname test.ovirt.org</code> which would be used to to change the hostname was giving weird errors inside the CI and the error traceback didn’t give much out apparently.</p>
<center><img src="/content/images/2017/06/dbus-error.png" /></center>
<p>I mean what does <code class="language-plaintext highlighter-rouge">Success</code> mean here? Quite the irony if you ask me. Hah</p>
<p>Searching around, didn’t result to much. Meanwhile Lukas thought there was some issue with the DBUS which was maybe due to some permissions change just before the particular task.</p>
<p>Lukas suggested that these tests were not specific to <code class="language-plaintext highlighter-rouge">ovirt</code> and that I try running the tests on a local setup. Had another 2gig linode server for testing but all the same. The tests failed with the same error again. Which suggested that this must be an issue with the command itself being run inside the docker container.</p>
<p>After some fooling around this issue, I remembered that even the <code class="language-plaintext highlighter-rouge">hostname</code> command can be used to change the name of the dev box temporarily.</p>
<p>Tried that and it worked!</p>
<center><img src="/content/images/2017/06/travis-pass-log.png" /></center>
<p>Another thing which was making the tests fail were the <code class="language-plaintext highlighter-rouge">ansible-lints</code>.</p>
<p>I had to replace <code class="language-plaintext highlighter-rouge">shell</code> module usage with <code class="language-plaintext highlighter-rouge">command</code> module in most of the places. Turns out that both these modules are just the same.</p>
<p>In the most use cases both modules lead to the same goal. Here are the main differences between these modules.</p>
<ul>
<li>With the Command module the command will be executed without being proceeded through a shell. As a consequence some variables like $HOME are not available. And also stream operations like <code class="language-plaintext highlighter-rouge"><</code>, <code class="language-plaintext highlighter-rouge">></code>, <code class="language-plaintext highlighter-rouge">|</code> and <code class="language-plaintext highlighter-rouge">&</code> will not work.</li>
<li>The Shell module runs a command through a shell, by default <code class="language-plaintext highlighter-rouge">/bin/sh</code>. This can be changed with the option executable. Piping and redirection are here therefor available.</li>
<li>The command module is more secure, because it will not be affected by the user’s environment.</li>
</ul>
<p>So by default, the <code class="language-plaintext highlighter-rouge">command</code> module is all that one requires to achieve most of things.</p>
<p>Also, there were some places in my PR where I had no choice but to use the <code class="language-plaintext highlighter-rouge">shell</code> module. For those, I had to pass the <code class="language-plaintext highlighter-rouge">skip_ansible_tag</code> for <code class="language-plaintext highlighter-rouge">ansible-lint</code> to skip those tasks.</p>
<p>And it’s always good to see your CI server mailing you things like this one.</p>
<center><img src="/content/images/2017/06/travis-pass.png" /></center>
<p>So far it’s been a great time for me and I have learned tons. Thanks for reading till this point.</p>
<p>Cheerio!</p>
<p><strong>EDIT:</strong></p>
<p><a href="https://medium.com/@ykaul/why-not-look-at-http-docs-ansible-com-ansible-hostname-module-html-to-change-the-host-name-b3c68f1481ec">Yaniv Kaul</a> suggested me that I try renaming the container using the http://docs.ansible.com/ansible/hostname_module.html module to rename the engine, but the module failed with the same DBUS error as the previous one.</p>
<p>Not a big change per se for testing it out.</p>
<div class="language-patch highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh">diff --git a/roles/ovirt-engine-rename/tasks/main.yml b/roles/ovirt-engine-rename/tasks/main.yml
index 8c96082..ed28298 100644
</span><span class="gd">--- a/roles/ovirt-engine-rename/tasks/main.yml
</span><span class="gi">+++ b/roles/ovirt-engine-rename/tasks/main.yml
</span><span class="p">@@ -49,7 +49,9 @@</span>
# to get the hostname, if not let it remain
- name: Changing the system host name to reflect the new engine-name
# command: hostnamectl set-hostname
<span class="gd">- command: hostname { { ovirt_engine_rename_new_fqdn } }
</span><span class="gi">+ # command: hostname { { ovirt_engine_rename_new_fqdn } }
+ hostname:
+ name: '{ { ovirt_engine_rename_new_fqdn } }'
</span> tags:
- skip_ansible_lint
</code></pre></div></div>
<p>Had chat with Lukas about it and he suggested that most of the systems use the DNS to pull out the hostname. So that task in the playbook ovirt-engine-rename would not be necessary and commented out on most cases.</p>
<p>Will be having a look at <a href="https://lago.readthedocs.io/en/latest/">Lago</a> and ovirt-system-test as you suggested as Yaniv Kaul suggested for end to end testing.</p>
<h2 id="older-blog-post">Older Blog post</h2>
<ul>
<li><a href="https://medium.com/@tasdikrahman/using-ansible-playbooks-to-install-ovirt-4-1-on-centos-7-linode-9dadd90143af">https://medium.com/@tasdikrahman/using-ansible-playbooks-to-install-ovirt-4-1-on-centos-7-linode-9dadd90143af</a></li>
<li><a href="https://medium.com/@tasdikrahman/installing-ovirt-4-1-on-centos-7-digitalocean-fbb4fa037c11">https://medium.com/@tasdikrahman/installing-ovirt-4-1-on-centos-7-digitalocean-fbb4fa037c11</a></li>
<li><a href="https://medium.com/@tasdikrahman/community-bonding-period-gsoc-2017-with-ovirt-org-5d0faac95257">https://medium.com/@tasdikrahman/community-bonding-period-gsoc-2017-with-ovirt-org-5d0faac95257</a></li>
<li><a href="https://medium.com/@tasdikrahman/hello-ovirt-gsoc-2017-6c18100f21dd">https://medium.com/@tasdikrahman/hello-ovirt-gsoc-2017-6c18100f21dd</a></li>
<li><a href="https://medium.com/@tasdikrahman/testing-your-ansible-roles-using-travis-ci-d186749a8edf">https://medium.com/@tasdikrahman/testing-your-ansible-roles-using-travis-ci-d186749a8edf</a></li>
</ul>
PyCon Taiwan 2017, Taipei
2017-06-12T00:00:00+00:00
https://www.tasdikrahman.com/2017/06/12/PyCon-Taiwan-2017-Taipei
<p>I have been going to PyCon India since the 2014 edition and have only missed PyCon Pune, the new regional PyCon that we have which was held at the start of this year. </p>
<p>Every year? Yes. But why? I dunno, maybe because I love python too much? Also partly because it’s the only time in the year where I get to meet all my friends in the Python community.</p>
<p>But this year was a little different.</p>
<p>This was the first year when I was going to a PyCon being held in a different country altogether and I would be lying if I said I was not excited about it.</p>
<p>Another reason to be excited was</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Will be speaking at <a href="https://twitter.com/PyConTW">@PyConTW</a>, 2017. Couldn't be more excited to meet you guys <a href="https://twitter.com/data__wizard">@data__wizard</a> <a href="https://twitter.com/uranusjr">@uranusjr</a> <a href="https://twitter.com/andrewgodwin">@andrewgodwin</a> <a href="https://twitter.com/freakboy3742">@freakboy3742</a> 😄 <a href="https://twitter.com/hashtag/pycontw?src=hash">#pycontw</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/870256422226862080">June 1, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<blockquote>
<p>Yes. This was the first time I would be speaking in PyCon! I was going to share the stage with some great developers and that was also a scary thought in itself.</p>
</blockquote>
<p>Crazy right? I know and I was pumped!</p>
<blockquote>
<p>I mean what if I fumble up, make some dumb mistake or what if I get a stage fear while presenting?</p>
</blockquote>
<p>Genuine questions but I was not thinking of any of the above when I started my trip from Bangalore</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">BLR ✈️ BKK ✈️ TPE. I thought jet lag wasn't a thing. I was wrong. <a href="https://twitter.com/hashtag/pycontw?src=hash">#pycontw</a> <a href="https://twitter.com/hashtag/python?src=hash">#python</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/872789142010122240">June 8, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<p>Travelling for a whole day straight does take toll on your body. And oh boy, the body aches!</p>
<h2 id="day-1">Day 1</h2>
<p>The keynote was given by <a href="http://www.csie.ntu.edu.tw/~htlin/">Hsuan-Tien Lin</a> who is working at appier right now. His talk revolved around the choices one should make while building an AI system.</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">It's a full house at <a href="https://twitter.com/PyConTW">@PyConTW</a> 🙌 <a href="https://twitter.com/hashtag/pycontw?src=hash">#pycontw</a> <a href="https://twitter.com/hashtag/day1?src=hash">#day1</a> <a href="https://t.co/mV2XTdtZ2L">pic.twitter.com/mV2XTdtZ2L</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/873214548534206465">June 9, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<p>This was followed by another keynote by <a href="https://speakerdeck.com/willingc/the-state-of-python-for-education">Carol Willing</a> which revolved around python for education highlighting recent innovations from Science, Data Science, web, and electronics (Raspberry Pi, MicroPython and CircuitPython) as well as a brief recap of innovative ideas from PyCon in Portland.</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">"The State of Python for Education" by our keynote speaker <a href="https://twitter.com/WillingCarol">@WillingCarol</a> 😄 <a href="https://twitter.com/hashtag/pycontw?src=hash">#pycontw</a> <a href="https://t.co/fGfP5BTSoT">pic.twitter.com/fGfP5BTSoT</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/873051990317608961">June 9, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<p>The talks I attended that day were <a href="https://tw.pycon.org/2017/events/talk/325227434406314071/">Python module in rust</a> and <a href="https://speakerdeck.com/dawny33/understanding-serverless-architectures">Understanding Serverless Architecture</a>. The rust talk was positioned more off as an intro to the language itself and how it was somewhat similar in terms of packaging and distribution of modules with Python. There are quite few more things which I noticed were very similar like Rust Lambda was very similar to Python’s lamda’s. It got real interesting when the speaker went into the details of how we could use rust code inside python just like how someone usually uses cffi, cpython or ctypes for achieving the same.</p>
<p>The server less talk delved into how one would leverage the new architectural styles. Which would be not having a server at all! Crazy stuff I would say and a great talk by my friend <a href="https://twitter.com/data__wizard/">Rohit</a></p>
<p>Day 1 ended with all the speakers and organisers for PyCon Taiwan heading to the speaker night being held at Courtyard Taipei. And the food was delightful.</p>
<h2 id="day-2">Day 2</h2>
<p><a href="https://twitter.com/andrewgodwin">Andrew Godwin</a> gave his keynote which revolved around a discussion about how most software is never designed with failure in mind, and the consequences can range from slightly annoying to life-threatening. He gave an insight about how the industry has already tackled failure incredibly effectively—aviation—and what lessons we can learn from it and apply to our own code both locally and distributed.</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Skynet? Nah. It's with us 😀 <a href="https://twitter.com/hashtag/pycontw?src=hash">#pycontw</a> <a href="https://t.co/OIkvXU1eZK">pic.twitter.com/OIkvXU1eZK</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/873053362765217792">June 9, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<p>Some other talks which I attended were <a href="https://www.slideshare.net/penvirus/global-interpreter-lock-episode-iii-cat-lt-devzero-gil">Global Interpreter Lock: Episode III - cat < /dev/zero > GIL</a> and <a href="https://www.slideshare.net/ssuser2cbb78/pycon-tw-2017-why-do-projects-fail-lets-talk-about-the-story-of-sinonpy">Why do projects fail? Let’s talk about the story of Sinon.PY</a></p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/data__wizard">@data__wizard</a> here at <a href="https://twitter.com/PyConTW">@PyConTW</a> with his Serverless talk. You go boy! 😀 <a href="https://twitter.com/hashtag/pycontw?src=hash">#pycontw</a> <a href="https://t.co/pySudXYnf4">pic.twitter.com/pySudXYnf4</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/873091601484070912">June 9, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<p>Later that day we had PyNight which started with a brief performance by my friend Adrian and he plays really good!</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">PyNight solo by Adrian over here. And this guy has some serious 🎹 skills I must say! <a href="https://twitter.com/hashtag/PyConTW?src=hash">#PyConTW</a> <a href="https://t.co/2QmBbmvZCg">pic.twitter.com/2QmBbmvZCg</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/873491014148685824">June 10, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<p>I finally got to have a chat with Andrew and Russell and it was a fanboy moment for me</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Had a great convo about OSS revenue models, personal projects, community building & Django Channels w/ <a href="https://twitter.com/andrewgodwin">@andrewgodwin</a> <a href="https://twitter.com/freakboy3742">@freakboy3742</a> <a href="https://twitter.com/hashtag/PyConTW?src=hash">#PyConTW</a> <a href="https://t.co/NgcQGRFQDn">pic.twitter.com/NgcQGRFQDn</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/873551798501482497">June 10, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<h2 id="day-3">Day 3</h2>
<p>As we headed to the last day of the conference, we started with a keynote by <a href="https://twitter.com/freakboy3742">Russell</a> where he discussed about the lessons we can learn from the past and present of open source communities, so that we can ensure we have a strong, healthy Python community going into the future.</p>
<p>The only talk I could attend that day was <a href="https://www.slideshare.net/masahitojp/the-benefits-of-type-hintss">“enjoy type hinting and its benefits”</a>.</p>
<p>I was mostly making some last minute changes to the slides for my talk slotted for the afternoon session.</p>
<center><img src="/content/images/2017/06/tasdik-pycontw-1.jpg" /></center>
<center><img src="/content/images/2017/06/tasdik-pycontw-2.jpg" /></center>
<blockquote>
<p>I was happy that I had finally given my talk! :)</p>
</blockquote>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Talk slides for my talk at <a href="https://twitter.com/PyConTW">@PyConTW</a> titled "Diving deep on how imports work in Python" <a href="https://t.co/i7UYC2oKWC">https://t.co/i7UYC2oKWC</a> <a href="https://twitter.com/hashtag/PyConTW?src=hash">#PyConTW</a> <a href="https://twitter.com/hashtag/Python?src=hash">#Python</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/874185974908977152">June 12, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
<p>And it was a wrap up for PyCon Taiwan 2017.</p>
<p>I had a great time and made amazing friends. Adrian, Tzu, Keith, Mike, Yahsin and I missing out a lot of names in this list.</p>
<p>You will be missed Taiwan!</p>
<center><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">So it's a wrap up for <a href="https://twitter.com/PyConTW">@PyConTW</a>. Had a great time with the community back there. Missing you guys already! TPE ✈️ BKK ✈️ BLR <a href="https://twitter.com/hashtag/PyConTW?src=hash">#PyConTW</a> <a href="https://t.co/2xlQd5Mwz8">pic.twitter.com/2xlQd5Mwz8</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/874049878371872768">June 11, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></center>
Implementing Role Based Access Control
2017-06-01T00:00:00+00:00
https://www.tasdikrahman.com/2017/06/01/Implementing-role-based-access-Control-easyrbac-python
<blockquote>
<p>You can find the python module for implementing RBAC0 here at <a href="https://github.com/tasdikrahman/easyrbac">https://github.com/tasdikrahman/easyrbac</a></p>
</blockquote>
<h2 id="main-idea-behind-it">Main Idea behind it</h2>
<p>If I have some 100 users in my system and for each user. I need to have some form of ACL using which the system makes choices whether they should be having authorisation for different actions on resources. Meaning, only the actors should be able to perform only those actions for which they are having authorisation.</p>
<p>How do you solve that?</p>
<p>Do you remember your English class ** <em>teacher</em> **? I sure do. She used to tell all kinds of interesting facts in and around Indian history. Anyways</p>
<p>Do you see, the word teacher here? What is that?</p>
<p>A Role? So whoever was a ** <em>teacher</em> ** had a <code class="language-plaintext highlighter-rouge">role</code> as a teacher in the school?</p>
<p>What permissions/privileges did they have on the resources? Were they needed to be as assigned special permissions/authorisations on per teacher basis(yes, for let’s say the CS teacher gets access to the CS labs at any time but that would be an exception). More or less they had a lot of responsibilities or so as to speak privileges common among themselves</p>
<p>So it would be common sense to group them (teachers) together and create an entity(role in this case).</p>
<p>How does this help?</p>
<p>Now instead of defining some 100 rules for 100 teachers, I can create a <code class="language-plaintext highlighter-rouge">Role</code> called <code class="language-plaintext highlighter-rouge">teacher</code> and create users which would be assigned the role of a teacher.</p>
<p>This way I can easily manage the permissions for all the 100 teachers without getting repetitive as now I would just need to edit rules at the role level and not the 100 Users which were assigned the role of a teacher.</p>
<p>I also get the freedom to easily delete a role from a user in a cleaner manner. Imagine writing individual ACL policies for every user out there. Horror right?</p>
<p>Analogous to this would be <code class="language-plaintext highlighter-rouge">iptables</code>, this model does not scale very well when you have a couple more of users in your system. By practicality, your existing rules would be circus managing which would be a huge man-hour consumer. This also, increases the chance of human error while doing so. Editing and removing some user from that? Even harder.</p>
<p>This is what <code class="language-plaintext highlighter-rouge">ufw</code> solves for you.</p>
<h2 id="introduction-to-rbac">Introduction to RBAC</h2>
<p>RBAC is Role Based Access Control, a powerful complement to traditional access control strategies and is the most manageable model (Who’s how to do what (Which)) and is (one of)the most popular access control mechanism which greatly reduces the workload of security administrators.</p>
<p>Here, the use of the role as an authorised intermediary, its basic idea is to access the permissions assigned to a certain role, the user by playing a different role to obtain the role of access rights have access</p>
<center><img src="/content/images/2017/06/rbac_model.jpg" /></center>
<p>Professor Sandhu has done a great job in explaining everything. Do check his article out which I have pointed at the end. </p>
<p>So I would use an RBAC for it’s</p>
<h1 id="advantages">Advantages</h1>
<ul>
<li>Easy to manage</li>
<li>Easy to classify according to work needs</li>
<li>To grant the minimum privilege</li>
</ul>
<h2 id="rbac-mainstream-model">RBAC mainstream model</h2>
<p>Layered RBAC basic model</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">RBAC0</code>: contains RBAC core part</li>
<li><code class="language-plaintext highlighter-rouge">RBAC1</code>: contains RBAC0, another role inheritance (RH)</li>
<li><code class="language-plaintext highlighter-rouge">RBAC2</code>: contains RBAC0, and other constraints (Constraints)</li>
<li><code class="language-plaintext highlighter-rouge">RBAC3</code>: Contains all levels of content and is a complete model</li>
</ul>
<h2 id="easyrbac">easyrbac</h2>
<p>Easiest way you can grok and retain all I had read, was to make something out of it. <code class="language-plaintext highlighter-rouge">easyrbac</code> was born out of it. I have tried implementing <code class="language-plaintext highlighter-rouge">RBAC0</code> for this release. The next release would focus on getting <code class="language-plaintext highlighter-rouge">RBAC1</code>, which includes role inheritance.</p>
<p><code class="language-plaintext highlighter-rouge">easyrbac</code> has a very simple API to interact around and create <code class="language-plaintext highlighter-rouge">Roles</code> and <code class="language-plaintext highlighter-rouge">Users</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">easyrbac</span> <span class="kn">import</span> <span class="n">Role</span><span class="p">,</span> <span class="n">User</span>
<span class="n">everyone_role</span> <span class="o">=</span> <span class="n">Role</span><span class="p">(</span><span class="s">'everyone'</span><span class="p">)</span>
<span class="n">admin_role</span> <span class="o">=</span> <span class="n">Role</span><span class="p">(</span><span class="s">'admin'</span><span class="p">)</span>
<span class="n">everyone_user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">roles</span><span class="o">=</span><span class="p">[</span><span class="n">everyone_role</span><span class="p">])</span>
<span class="n">admin_user</span> <span class="o">=</span> <span class="n">User</span><span class="p">(</span><span class="n">roles</span><span class="o">=</span><span class="p">[</span><span class="n">admin_role</span><span class="p">,</span> <span class="n">everyone_role</span><span class="p">])</span>
</code></pre></div></div>
<p>For User resource access permissions allocation</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">acl</span> <span class="o">=</span> <span class="n">AccessControlList</span><span class="p">()</span>
<span class="n">acl</span><span class="p">.</span><span class="n">resource_read_rule</span><span class="p">(</span><span class="n">everyone_role</span><span class="p">,</span> <span class="s">'GET'</span><span class="p">,</span> <span class="s">'/api/v1/employee/1/info'</span><span class="p">)</span>
<span class="n">acl</span><span class="p">.</span><span class="n">resource_delete_rule</span><span class="p">(</span><span class="n">admin_role</span><span class="p">,</span> <span class="s">'DELETE'</span><span class="p">,</span> <span class="s">'/api/v1/employee/1/'</span><span class="p">)</span>
<span class="c1"># checking READ operation on resource for user `everyone_user`
</span><span class="k">for</span> <span class="n">user_role</span> <span class="ow">in</span> <span class="p">[</span><span class="n">role</span><span class="p">.</span><span class="n">get_name</span><span class="p">()</span> <span class="k">for</span> <span class="n">role</span> <span class="ow">in</span> <span class="n">everyone_user</span><span class="p">.</span><span class="n">get_roles</span><span class="p">()]:</span>
<span class="k">assert</span> <span class="n">acl</span><span class="p">.</span><span class="n">is_read_allowed</span><span class="p">(</span><span class="n">user_role</span><span class="p">,</span> <span class="s">'GET'</span><span class="p">,</span> <span class="s">'/api/v1/employee/1/info'</span><span class="p">)</span> <span class="o">==</span> <span class="bp">True</span>
<span class="c1"># checking WRITE operation on resource for user `everyone_user`
# Since you have not defined the rule for the particular, it will disallow any such operation by default.
</span><span class="k">for</span> <span class="n">user_role</span> <span class="ow">in</span> <span class="p">[</span><span class="n">role</span><span class="p">.</span><span class="n">get_name</span><span class="p">()</span> <span class="k">for</span> <span class="n">role</span> <span class="ow">in</span> <span class="n">everyone_user</span><span class="p">.</span><span class="n">get_roles</span><span class="p">()]:</span>
<span class="k">assert</span> <span class="n">acl</span><span class="p">.</span><span class="n">is_write_allowed</span><span class="p">(</span><span class="n">user_role</span><span class="p">,</span> <span class="s">'WRITE'</span><span class="p">,</span> <span class="s">'/api/v1/employee/1/info'</span><span class="p">)</span> <span class="o">==</span> <span class="bp">False</span>
<span class="c1"># checking WRITE operation on resource for user `admin_user`
</span><span class="k">for</span> <span class="n">user_role</span> <span class="ow">in</span> <span class="p">[</span><span class="n">role</span><span class="p">.</span><span class="n">get_name</span><span class="p">()</span> <span class="k">for</span> <span class="n">role</span> <span class="ow">in</span> <span class="n">everyone_user</span><span class="p">.</span><span class="n">get_roles</span><span class="p">()]:</span>
<span class="k">if</span> <span class="n">user_role</span> <span class="o">==</span> <span class="s">'admin'</span><span class="p">:</span> <span class="c1"># as a user can have more than one role assigned to them
</span> <span class="k">assert</span> <span class="n">acl</span><span class="p">.</span><span class="n">is_delete_allowed</span><span class="p">(</span><span class="n">user_role</span><span class="p">,</span> <span class="s">'DELETE'</span><span class="p">,</span> <span class="s">'/api/v1/employee/1/'</span><span class="p">)</span> <span class="o">==</span> <span class="bp">True</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">assert</span> <span class="n">acl</span><span class="p">.</span><span class="n">is_delete_allowed</span><span class="p">(</span><span class="n">user_role</span><span class="p">,</span> <span class="s">'DELETE'</span><span class="p">,</span> <span class="s">'/api/v1/employee/1/'</span><span class="p">)</span> <span class="o">==</span> <span class="bp">False</span>
</code></pre></div></div>
<h2 id="future-work">Future work</h2>
<ul>
<li>Adding hierarchical roles, which represent parent<->child relations</li>
<li>Adding this on top of Bottle/Flask</li>
</ul>
<h2 id="literature-material">Literature material</h2>
<ul>
<li><a href="http://profsandhu.com/articles/advcom/adv_comp_rbac.pdf">http://profsandhu.com/articles/advcom/adv_comp_rbac.pdf</a></li>
<li><a href="http://www.comp.nus.edu.sg/~tankl/cs5322/readings/rbac1.pdf">http://www.comp.nus.edu.sg/~tankl/cs5322/readings/rbac1.pdf</a></li>
</ul>
<h2 id="github-repo">Github repo</h2>
<ul>
<li><a href="https://github.com/tasdikrahman/easyrbac">https://github.com/tasdikrahman/easyrbac</a></li>
</ul>
<p>Cheerio</p>
Learnings from analyzing my compromised server (Linode)
2017-05-25T00:00:00+00:00
https://www.tasdikrahman.com/2017/05/25/Learnings-from-analyzing-my-compromised-server-linode-Google-Summer-of-Code-oVirt-2017
<blockquote>
<p>DISCLAIMER: All views presented are personal and not that of my employers or anyone else for that matter. In no occasion do I blame Linode for this security breach. It was because I did not follow the best practices which you will read and not repeat again.</p>
</blockquote>
<p>Yesterday, I was having a great day!</p>
<p>I had the daily goal of walking 6000 steps done. Thanks to my swanky new Mi band 1 (bade goodbye to my Sony SWR10). Wrote about installing <code class="language-plaintext highlighter-rouge">ovirt-engine</code> using an ansible-playbook in a post <a href="http://www.tasdikrahman.com/2017/05/24/Installing-ovirt-4.1-on-centos-7-using-ansible-linode-Google-Summer-of-Code-oVirt-2017/">yesterday</a>. My mom wasn’t telling me for a change to get a haircut and suggested I start packing some proper clothes (she means washed and ironed) for my upcoming <a href="https://tw.pycon.org/2017/en-us/events/talk/342865744498786414/">trip to Taiwan</a> which would be my first international trip.</p>
<p>You need some getting used to, to the event of waking up to your significant others video call early in the morning. And I tell you if you had some bad sleep like me yesterday. You would hardly be in the mood for it. Sorry sunshine.</p>
<p>Anyways. So about yesterday, I finally decided to shift from DigitalOcean to Linode. As evident from my tweet yesterday.</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/tasdikrahman">@tasdikrahman</a> <a href="https://twitter.com/linode">@linode</a> Now, I'm not trying to entice you further, but you could use the code 'linode10' and get some credit to start with. It's nice to have.</p>— Feeling OK (@FeelingSohSoh) <a href="https://twitter.com/FeelingSohSoh/status/867382204430766080">May 24, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Thanks for the credits Linode. Appreciate it :)</p>
<p>Fast forward some hours. I have a 4GB centOS 7 box up and running on a Singapore datecenter. After some failed attempts, got my <code class="language-plaintext highlighter-rouge">ansible-playbook</code> to run on the remote machine which installed ovirt-engine on it.</p>
<h2 id="happy-period-quickly-turning-to-a-bad-one">Happy period quickly turning to a bad one</h2>
<p>Everything is fine and dandy and I am watching a basketball match with my friends after office hours. 4 minutes left to the final whistle, Cracking match on, everyone is playing like a pro. A very close call between the two teams, but one gets the better of the other one.</p>
<p>Returning back home. I get this buzz on my phone. Turns out it’s an email from Linode. Daym. I thought was I billed already?</p>
<center><img src="/content/images/2017/05/outbound-network-linode-mail.png" /></center>
<p>Trust me on this, I was really not sure what to do of this for the first two minutes when I read the email.</p>
<p>I opened the Linode admin panel to check out what was my server up to. And the CPU graph had jumped off the hooks.</p>
<center><img src="/content/images/2017/05/CPU-chart-linode-1.png" /></center>
<p>Same was the case with the network graph</p>
<center><img src="/content/images/2017/05/network-chat-linode-1.png" /></center>
<p>Looking at the network log’s suggested a high amount of outbound traffic coming from my server, further cementing the Linode support ticket that I got.</p>
<center><img src="/content/images/2017/05/network-io-linode-1.png" /></center>
<p>I ssh’d inside my server to see what was going on.</p>
<center><img src="/content/images/2017/05/failed-ssh-tries.png" /></center>
<p>I will be damned. I don’t remember sleep typing my password continuously for that long!</p>
<p>Let me tell you, you don’t do a <code class="language-plaintext highlighter-rouge">cat /var/log/secure</code> at this point as the file would just be spit continously at you with no end of stopping.</p>
<p>Did <code class="language-plaintext highlighter-rouge">head</code> (even a <code class="language-plaintext highlighter-rouge">tail</code> can do) to it. Going through the start of the file, everything was fine until I started to see the extremely less epoch time between two failed attempts. This confirmed my hunch that some script kiddie was trying to brute force through the <code class="language-plaintext highlighter-rouge">root</code> user login.</p>
<center><img src="/content/images/2017/05/var-log-secure-failed-attempts.png" /></center>
<p>I know, I should have disabled root login at the start and used ssh-keys to access my server. But I just delayed it to be done the next day. My fault.</p>
<blockquote>
<p>The logical thing now would be to start <code class="language-plaintext highlighter-rouge">iptables</code> (or) <code class="language-plaintext highlighter-rouge">ufw</code> and block outbound traffic as well as inbound except required stuff. Then take all the logs and look at them.</p>
</blockquote>
<h2 id="yum-install-breaks">yum install breaks</h2>
<center><img src="/content/images/2017/05/ufw-notfound.png" /></center>
<p>Now I am pretty sure most linux distrbutions, Ubuntu for example ships with <code class="language-plaintext highlighter-rouge">ufw</code> by default. An <code class="language-plaintext highlighter-rouge">SELinux</code> like <code class="language-plaintext highlighter-rouge">centOS</code> not shipping <code class="language-plaintext highlighter-rouge">ufw</code> cannot be remotely possible. My hunch was that the perpetrator must have removed it altogether.</p>
<p>No problemo, I could do a <code class="language-plaintext highlighter-rouge">yum install ufw</code> right?</p>
<center><img src="/content/images/2017/05/yum-timeout.png" /></center>
<p>I try installing it and there are constant timeouts on the network calls being made by the server to the mirrors holding the package. Same is the case with other packages like <code class="language-plaintext highlighter-rouge">strace</code>, <code class="language-plaintext highlighter-rouge">tcpdump</code> et al.</p>
<center><img src="/content/images/2017/05/yum-timeout-1.png" /></center>
<p>The connection is really sluggish even though I am on a network having very less latency.</p>
<h2 id="when-ufw-fails-go-back-to-good-old-iptables">when <code class="language-plaintext highlighter-rouge">ufw</code> fails go back to good old <code class="language-plaintext highlighter-rouge">iptables</code></h2>
<p><code class="language-plaintext highlighter-rouge">ufw</code> stands for <strong>uncomplicated firewall</strong>. So that you don’t have to directly deal with the low level intricacies</p>
<p>With <code class="language-plaintext highlighter-rouge">ufw</code>, for blocking all incoming traffic from a given IP address would have been as simple as doing a</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>ufw deny from <ip-address>
</code></pre></div></div>
<p>But since I don’t have it, falling back to <code class="language-plaintext highlighter-rouge">iptables</code>.</p>
<center><img src="/content/images/2017/05/iptables-drop-ip.png" /></center>
<h2 id="checking-all-outgoing-requestsconnections-from-the-server">Checking all outgoing requests/connections from the server</h2>
<p>For that I did a <code class="language-plaintext highlighter-rouge">$ netstat -nputw</code></p>
<p>lists all <code class="language-plaintext highlighter-rouge">UDP (u)</code>, <code class="language-plaintext highlighter-rouge">TCP (t)</code> and <code class="language-plaintext highlighter-rouge">RAW (w)</code> outgoing connections (not using l or a) in a numeric form (n, prevents possible long-running DNS queries) and includes the program (p) associated with that.</p>
<center><img src="/content/images/2017/05/netstat.png" /></center>
<p>What has a <code class="language-plaintext highlighter-rouge">cd</code> command got to do with making network connections?</p>
<p>Did a <code class="language-plaintext highlighter-rouge">$ ps aux</code> to check all the system processes and the thing was standing out here too</p>
<center><img src="/content/images/2017/05/ps-aux-cd.png" /></center>
<p>It was taking up 95% of the whole CPU! When was the last time you saw a <code class="language-plaintext highlighter-rouge">coreutil</code> doing that?</p>
<p>Digging down further, I wanted to see what were the files being opened by our, at this point I may say modified <code class="language-plaintext highlighter-rouge">cd</code> program</p>
<center><img src="/content/images/2017/05/lsof-cd.png" /></center>
<p>The perpetrator had been running his malicious program under <code class="language-plaintext highlighter-rouge">/usr/bin</code> which obviously meant he did gain root access to my server. There was no other way I could think of, through which the perpetrator could have placed it under <code class="language-plaintext highlighter-rouge">/usr/bin</code> otherwise.</p>
<p>If you feel, someone else too is logged in, you can easily check that by doing a</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>netstat <span class="nt">-nalp</span> | <span class="nb">grep</span> “:22″
</code></pre></div></div>
<p>To double check that he did get in, I ran a</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat</span> /var/log/secure<span class="k">*</span> | <span class="nb">grep </span>ssh | <span class="nb">grep </span>Accept<span class="sb">`</span>
</code></pre></div></div>
<p>And not so surprisingly,</p>
<center><img src="/content/images/2017/05/he-got-in.png" /></center>
<p>At this point I don’t have anything like <code class="language-plaintext highlighter-rouge">strace</code> or <code class="language-plaintext highlighter-rouge">gdb</code> to it to see what this program is doing, as mentioned earlier the inability do install anything using <code class="language-plaintext highlighter-rouge">yum</code> through the network.</p>
<p>Compiling from source?</p>
<p><code class="language-plaintext highlighter-rouge">wget</code> was no where to be found!</p>
<p>I could only think of killing the process once and see what happens then. I do a <code class="language-plaintext highlighter-rouge">$ kill -9 3618</code> to kill the process and check <code class="language-plaintext highlighter-rouge">ps aux</code> again looking for the particular program only to find it back again as a new process</p>
<h2 id="conjuring-3">Conjuring 3</h2>
<center><img src="/content/images/2017/05/cd-resurfaces.png" /></center>
<p>Again killing it and checking the processes. This time I don’t see it.</p>
<p>But I was wrong, looks like there is another process hogging CPU in a similar fashion like <code class="language-plaintext highlighter-rouge">cd</code> was doing.</p>
<center><img src="/content/images/2017/05/resurface-as-ifconfig.png" /></center>
<p>Tbh, this was bat shoot crazy stuff for me. And I was feeling the adrenaline pump. Sleep was something I lost completely even though it was some 3a.m. in the morning. This crap was way too exciting to be left over to be done in the morning!</p>
<p>I check <code class="language-plaintext highlighter-rouge">ps aux</code> twice just to double check if there anything in the lines of what happened. And there is nothing this time to send a <code class="language-plaintext highlighter-rouge">SIG KILL</code> to.</p>
<h2 id="relief">Relief?</h2>
<p>I try installing packages through <code class="language-plaintext highlighter-rouge">yum</code> and it’s the same old story of network-timeout.</p>
<p>You might also be wondering why I did not check <code class="language-plaintext highlighter-rouge">bash history</code>. Trust me, it was clean! I could only see the things which I had typed away. Whatever it was, it was smart. But I guess this is just the basic stuff of not leaving anything behind.</p>
<h2 id="nmap-away-all-the-things">nmap away all the things</h2>
<p>Cause why not?</p>
<p>Ran a quick scan on the server using</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>nmap ovirtlinode.gsoc.org
Starting Nmap 7.40 <span class="o">(</span> https://nmap.org <span class="o">)</span> at 2017-05-25 14:35 IST
Nmap scan report <span class="k">for </span>ovirtlinode.gsoc.org
Host is up <span class="o">(</span>0.035s latency<span class="o">)</span><span class="nb">.</span>
Not shown: 978 filtered ports
PORT STATE SERVICE
22/tcp open ssh
25/tcp closed smtp
80/tcp open http
113/tcp closed ident
179/tcp closed bgp
443/tcp open https
1723/tcp closed pptp
2000/tcp closed cisco-sccp
2222/tcp open EtherNetIP-1
5432/tcp open postgresql
6000/tcp closed X11
6001/tcp closed X11:1
6002/tcp closed X11:2
6003/tcp closed X11:3
6004/tcp closed X11:4
6005/tcp closed X11:5
6006/tcp closed X11:6
6007/tcp closed X11:7
6009/tcp closed X11:9
6025/tcp closed x11
6059/tcp closed X11:59
6100/tcp open synchronet-db
</code></pre></div></div>
<p>Checking the UDP connections</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>nmap <span class="nt">-sS</span> <span class="nt">-sU</span> <span class="nt">-T4</span> <span class="nt">-A</span> <span class="nt">-v</span> ovirtlinode.gsoc.org
Starting Nmap 7.40 <span class="o">(</span> https://nmap.org <span class="o">)</span> at 2017-05-25 14:37 IST
NSE: Loaded 143 scripts <span class="k">for </span>scanning.
NSE: Script Pre-scanning.
Initiating NSE at 14:37
Completed NSE at 14:37, 0.00s elapsed
Initiating NSE at 14:37
Completed NSE at 14:37, 0.00s elapsed
Initiating Ping Scan at 14:37
Scanning ovirtlinode.gsoc.org <span class="o">(</span>172.104.36.181<span class="o">)</span> <span class="o">[</span>4 ports]
Completed Ping Scan at 14:37, 0.01s elapsed <span class="o">(</span>1 total hosts<span class="o">)</span>
Initiating SYN Stealth Scan at 14:37
Scanning ovirtlinode.gsoc.org <span class="o">(</span>172.104.36.181<span class="o">)</span> <span class="o">[</span>1000 ports]
Discovered open port 80/tcp on 172.104.36.181
Discovered open port 443/tcp on 172.104.36.181
Discovered open port 22/tcp on 172.104.36.181
Discovered open port 6100/tcp on 172.104.36.181
Discovered open port 5432/tcp on 172.104.36.181
Discovered open port 2222/tcp on 172.104.36.181
Completed SYN Stealth Scan at 14:37, 4.22s elapsed <span class="o">(</span>1000 total ports<span class="o">)</span>
Initiating UDP Scan at 14:37
Scanning ovirtlinode.gsoc.org <span class="o">(</span>172.104.36.181<span class="o">)</span> <span class="o">[</span>1000 ports]
Completed UDP Scan at 14:37, 4.19s elapsed <span class="o">(</span>1000 total ports<span class="o">)</span>
Initiating Service scan at 14:37
</code></pre></div></div>
<p>Did not get much out of it.</p>
<p>So this brings me to the moment where I see the attack vector which was placed by the perpetrator becomes dormant. I did not see any other activity which would wry my attention again.</p>
<p>I let my server run for the night (stupid call but I was just plain old curious) to check in the morning what was the status.</p>
<p>Turned out there still had been a fair amount of outbound calls being made during that time.</p>
<center><img src="/content/images/2017/05/still-outboung-io.png" /></center>
<p>The CPU graph resonated the same</p>
<center><img src="/content/images/2017/05/cpu-graph-morn.png" /></center>
<p>I turned off the server for a brief period of time after this. The CPU graphs can be seen below during that period.</p>
<center><img src="/content/images/2017/05/final-cpu-graph.png" /></center>
<p>There were no outbound network calls after that.</p>
<center><img src="/content/images/2017/05/final-network-io.png" /></center>
<h2 id="aftermath">Aftermath</h2>
<p>I turned off the server for good. There’s no going back to it. I will be rebuilding the image to it.</p>
<p>It happened to have my public key on the server. This is not something one should immediately worry about. Going from public key to private key is exceptionally hard; that’s how public key cryptography works. (By “exceptionally” I mean that it’s designed to be resistant to well-funded government efforts; if it keeps the NSA from cracking you, it’ll be sure good enough for stopping your average joe)</p>
<p>But just to be on the safer side, I regenerated my ssh keys, deleted the old ones at the places it was being used and updated it with the new ones.</p>
<p>Just to be extra sure of everything, I checked the activity logs of the services which did use the older ssh keys. No unusual activities.</p>
<center><img src="/content/images/2017/05/github-logs.png" /></center>
<h2 id="learnings">Learnings</h2>
<ul>
<li>disable root password login</li>
</ul>
<p>The very first thing that I should have done after provisioning the server would have been to do this. This would infact have stopped the perpetrator from logging in as root and cause any havoc</p>
<ul>
<li>obfuscate the port <code class="language-plaintext highlighter-rouge">sshd</code> uses</li>
</ul>
<p>change it to something not common as the default <code class="language-plaintext highlighter-rouge">22</code> would be known by you as well as the perpetrator. ~As someone rightfully said, security lies in obscurity.~</p>
<p>But thinking that security through obscurity is makes you fail safe. Think again. </p>
<p>Security by obscurity is a beginner fail! one thing is to keep secrets and make it harder to exploit, the other is to rely on secrets for security. The financials were doing it for long until they learned it does not work.</p>
<p>It’s better to use known good practices, protocols and techniques than to come with your own, reinventing the wheel and trying to keep it secret. Reverse engineering has been done for everything, from space rockets to smart toasters. It takes usually 30min to fingerprint an OS version aling with libraries no matter how you protect it. Just looking at a ping RTT can identify the OS, for example.</p>
<p>Finally, nobody ever got blamed for using best practices. Yet, if you try to outsmart the system it’ll all be your fault.</p>
<ul>
<li>
<p>Use your public key to ssh into the machine instead of password login as suggested.</p>
</li>
<li>
<p>Take regular backups/snapshot’s of your server. That way if something funny does happen. You can always restore it to a previous state.</p>
</li>
<li>
<p>I kept a really easy to guess, dictionary based password. I have to admit it, this is simply the stupidest thing one can do. And yes, I did it. My only reasoning for doing that was that this was a throwaway server, but that makes up for no excuse for not following security practices.</p>
</li>
</ul>
<p>KEEP A STRONG PASSWORD! Even though there is a brute force attack on your server, it is relatively very hard to crack the password if you keep a strong one.</p>
<p>Bruce <a href="http://www.schneier.com/essay-148.html">has a nice essay</a> about the subject here. Won’t repeat what he has said so take a look at it.</p>
<p>Research suggests that adding password complexity requirements like upper case/numbers/symbols cause users to make easy to predict changes and cause them to create simpler passwords overall due to being harder to remember.</p>
<p>Also read <a href="https://www.owasp.org/index.php/Authentication_Cheat_Sheet#Implement_Proper_Password_Strength_Controls">OWASP’s blog</a> here about what they have to say about passwords</p>
<p>There was a discussion on <a href="https://security.stackexchange.com/questions/29836/what-are-good-requirements-for-a-password">security exchange</a> too over this</p>
<ul>
<li>protect ssh with fail2ban</li>
</ul>
<p>Fail2ban can mitigate this problem by creating rules that automatically alter your iptables firewall configuration based on a predefined number of unsuccessful login attempts. This will allow your server to respond to illegitimate access attempts without intervention from you.</p>
<ul>
<li>Some Files which are in the Common Attack Points:</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">ls</span> /tmp <span class="nt">-la</span>
<span class="nv">$ </span><span class="nb">ls</span> /var/tmp <span class="nt">-la</span>
<span class="nv">$ </span><span class="nb">ls</span> /dev/shm <span class="nt">-la</span>
</code></pre></div></div>
<p>Check these to have a look for something which should not be there.</p>
<p>Unluckily, I had already rebooted my server at that point which caused me to lose this info.</p>
<ul>
<li>
<p>Keep a pristine copy of critical system files (such as ls, ps, netstat, md5sum) somewhere, with an md5sum of them, and compare them to the live versions regularly. Rootkits will invariably modify these files. Use these copies if you suspect the originals have been compromised.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">aide</code> or <code class="language-plaintext highlighter-rouge">tripwire</code> will tell you of any files that have been modified - assuming their databases have not been tampered with.
Configure syslog to send your logfiles to a remote log server where they can’t be tampered with by an intruder. Watch these remote logfiles for suspicious activity</p>
</li>
<li>read your logs regularly - use <code class="language-plaintext highlighter-rouge">logwatch</code> or <code class="language-plaintext highlighter-rouge">logcheck</code> to synthesize the critical information.</li>
<li>Know your servers. Know what kinds of activities and logs are normal.</li>
<li>using tools like <a href="http://www.chkrootkit.org/download/">chkrootkit</a> to check for rootkits regularly</li>
</ul>
<h2 id="closing-notes">Closing notes</h2>
<blockquote>
<p>How do you know if your Linux server has been hacked?</p>
</blockquote>
<p>You don’t!</p>
<p>I know, I know — but it’s the paranoid, sad truth, really ;) There are plenty of hints of course, but if the system was targeted specifically — it might be impossible to tell. It’s good to understand that nothing is ever completely secure.</p>
<p>If your system was compromised, meaning once someone has root access on your host, you cannot trust anything you see because above and beyond the more obvious methods like modifying ps, ls, etc, one can simply attack kernel level system calls to subjugate IO itself. None of your system tools can be trusted to reveal the truth.</p>
<p>Simply put, you cant trust anything your terminal tells you, period.</p>
<p><strong>I would like to thank everyone at Linode. Their effort and dedication to maintaining our servers and their security, while ensuring an affordable price, enables us to succeed. Thanks guys</strong></p>
<p><strong>If I have made any mistakes or you think I should have done something else from what I have tried or not described something as it should be. Please feel free to point it out. I am just starting out on the infosec scene :)</strong></p>
<p>Cheers and stay secured!</p>
<p>And I just have to post this image.</p>
<center><img src="/content/images/2017/05/today-was-a-good-day-meme.jpg" /></center>
<blockquote>
<p>Catch the discussion over <a href="https://news.ycombinator.com/item?id=14416982">HN</a></p>
</blockquote>
<h2 id="further-read">Further read</h2>
<ul>
<li><a href="https://www.webair.com/community/can-check-server-hacked/">https://www.webair.com/community/can-check-server-hacked/</a></li>
<li><a href="https://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server">https://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server</a></li>
<li><a href="https://security.stackexchange.com/questions/7443/how-do-you-know-your-server-has-been-compromised">https://security.stackexchange.com/questions/7443/how-do-you-know-your-server-has-been-compromised</a></li>
<li><a href="https://wiki.centos.org/HowTos/Network/IPTables">https://wiki.centos.org/HowTos/Network/IPTables</a></li>
<li><a href="https://stackoverflow.com/questions/884525/how-to-investigate-what-a-process-is-doing">https://stackoverflow.com/questions/884525/how-to-investigate-what-a-process-is-doing</a></li>
<li><a href="https://jorge.fbarr.net/2014/01/19/introduction-to-strace/">https://jorge.fbarr.net/2014/01/19/introduction-to-strace/</a></li>
<li><a href="https://stackoverflow.com/questions/3972534/dsa-what-can-a-hacker-do-with-just-a-public-key">https://stackoverflow.com/questions/3972534/dsa-what-can-a-hacker-do-with-just-a-public-key</a></li>
<li><a href="https://security.stackexchange.com/questions/44094/isnt-all-security-through-obscurity">https://security.stackexchange.com/questions/44094/isnt-all-security-through-obscurity</a></li>
<li><a href="https://stackoverflow.com/questions/533965/why-is-security-through-obscurity-a-bad-idea">https://stackoverflow.com/questions/533965/why-is-security-through-obscurity-a-bad-idea</a></li>
</ul>
Using Ansible Playbooks to Install oVirt 4.1 on centOS 7 (Linode)
2017-05-24T00:00:00+00:00
https://www.tasdikrahman.com/2017/05/24/Installing-ovirt-4.1-on-centos-7-using-ansible-linode-Google-Summer-of-Code-oVirt-2017
<p>In my <a href="http://www.tasdikrahman.com/2017/05/21/Installing-ovirt-4.1-on-centos-7-digitalocean-Google-Summer-of-Code-oVirt-2017/">previous post</a>, I played around how to install <code class="language-plaintext highlighter-rouge">ovirt-engine</code> on a remote VM.</p>
<p>Turns out we can automate the whole process using ansible playbooks!</p>
<center><img src="/content/images/2017/05/060_net_00_featured.jpg" /></center>
<p>Secondly, I have thought of shifting from <a href="https://digitalocean.com/">digitalocean</a> to <a href="https://linode.com">linode</a>. Why?</p>
<p>Well, first off. It’s not that I don’t like digitalocean. It’s just that the prices for the VMS that I’am provisioning, are getting too high for me.</p>
<p>For a 4GB VM with 2 cores and 60GB of SSD to spare with. I am getting some 4TB of network I/O. 4TB for me is generous. All this adds up to a damage of $40/month or roughly speaking, $0.060/hour.</p>
<p>If you compare this to that of Linode’s offerings. I am getting the same 4GB VM with 48GB SSD Storage. 2 CPU Cores and 3TB XFER. This costs $20/mo or (.03/hr)</p>
<p>That’s like half of what I was paying to digitalocean for my servers! But the ease of use for digitalocean is pretty good. The only reason that I see if I want to use digitalocean for my VM’s would be if I wanted low latency. As the closest datecenter to me for linode is in Singapore. While we digitalcoean has a datacenter around in bangalore.</p>
<p>Enough of me ranting around.</p>
<h2 id="show-me-the-code">Show me the Code</h2>
<p>The playbook is already up there on <a href="https://github.com/rhevm-qe-automation/ovirt-ansible/">github</a> which holds the ansible roles for quite a many things.</p>
<p>Having a quick look at them</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">ls</span> <span class="nt">-la</span> roles
total 0
drwxr-xr-x 13 tasdik tasdik 442 May 23 14:28 <span class="nb">.</span>
drwxr-xr-x 20 tasdik tasdik 680 May 24 11:05 ..
<span class="nt">-rw-r--r--</span> 1 tasdik tasdik 0 May 23 14:28 ansible.cfg
drwxr-xr-x 6 tasdik tasdik 204 May 23 14:28 ovirt-collect-logs
drwxr-xr-x 6 tasdik tasdik 204 May 23 14:28 ovirt-common
drwxr-xr-x 6 tasdik tasdik 204 May 23 14:28 ovirt-engine-backup
drwxr-xr-x 7 tasdik tasdik 238 May 23 14:28 ovirt-engine-cleanup
drwxr-xr-x 6 tasdik tasdik 204 May 23 14:28 ovirt-engine-config
drwxr-xr-x 6 tasdik tasdik 204 May 23 14:28 ovirt-engine-install-packages
drwxr-xr-x 7 tasdik tasdik 238 May 23 14:28 ovirt-engine-remote-db
drwxr-xr-x 7 tasdik tasdik 238 May 23 14:28 ovirt-engine-setup
drwxr-xr-x 7 tasdik tasdik 238 May 23 14:28 ovirt-guest-agent
drwxr-xr-x 6 tasdik tasdik 204 May 23 14:28 ovirt-iso-uploader-conf
</code></pre></div></div>
<p>I would be showing around the role <code class="language-plaintext highlighter-rouge">ovirt-engine-setup</code> in this post.</p>
<p>Provision your server and <code class="language-plaintext highlighter-rouge">ssh</code> into it.</p>
<p>I have created an entry on my <code class="language-plaintext highlighter-rouge">/etc/hosts</code> on my local dev box for the VM where I am going to install <code class="language-plaintext highlighter-rouge">ovirt-engine</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat</span> /etc/hosts | <span class="nb">grep </span>gsoc
xxx.xxx.xx.xxx ovirtlinode.gsoc.org
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">xx.xxx.xx.xxx</code> would be the public IP of your server.</p>
<p>You have to change the <code class="language-plaintext highlighter-rouge">hostname</code> of your server. Will explaing the why in a bit.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ssh root@ovirtlinode.gsoc.org
root@ovirtlinode.gsoc.org<span class="s1">'s password:
[root@ovirtlinode ~]# hostnamectl set-hostname ovirtlinode.gsoc.org
[root@ovirtlinode ~]# hostname
ovirtlinode.gsoc.org
[root@ovirtlinode ~]#
</span></code></pre></div></div>
<p>Now from your local dev box</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git clone https://github.com/rhevm-qe-automation/ovirt-ansible/
<span class="nv">$ </span><span class="nb">cd </span>ovirt-ansible/
<span class="nv">$ </span><span class="nb">touch </span>site.yml inventory
</code></pre></div></div>
<p>The contents of the file above should be in the lines of</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>inventory
<span class="o">[</span>all:vars]
<span class="nv">ovirt_engine_type</span><span class="o">=</span>ovirt-engine
<span class="nv">ovirt_engine_version</span><span class="o">=</span>4.1
<span class="c"># Make sure that link to release rpm is working!!!</span>
<span class="nv">ovirt_rpm_repo</span><span class="o">=</span>http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm
<span class="nv">ovirt_engine_organization</span><span class="o">=</span>ovirtlinode.gsoc.org
<span class="nv">ovirt_engine_admin_password</span><span class="o">=</span>secret
<span class="o">[</span>engine]
ovirtlinode.gsoc.org <span class="nv">ansible_ssh_user</span><span class="o">=</span>root <span class="nv">ansible_ssh_pass</span><span class="o">=</span>secretpassword
<span class="err">$</span>
<span class="nv">$ </span><span class="nb">cat </span>site.yml
<span class="nt">---</span>
- hosts: engine
roles:
- role: ovirt-common
- role: ovirt-engine-install-packages
- role: ovirt-engine-setup
</code></pre></div></div>
<p>A quick note on the variable <code class="language-plaintext highlighter-rouge">ansible_ssh_pass=secretpassword</code>. This is not adviced! Don’t do this as now obviously your ssh password is
stored in a text file which can be read by anyone. And god forbid if you add this file to version control accidentally and push it.</p>
<p>Rule of thumb is to always use ssh keys to run playbooks!</p>
<p>After this, we need to check the ansible fact <code class="language-plaintext highlighter-rouge">ansible_fqdn</code>. This one is important, as the we will be accessing the Admin panel of
ovirt-engine using the FQDN of this value. Otherwise our ansible playbook won’t work.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible <span class="nt">-m</span> setup <span class="nt">-i</span> inventory engine <span class="nt">-a</span> <span class="s1">'filter=ansible_fqdn'</span>
ovirtlinode.gsoc.org | SUCCESS <span class="o">=></span> <span class="o">{</span>
<span class="s2">"ansible_facts"</span>: <span class="o">{</span>
<span class="s2">"ansible_fqdn"</span>: <span class="s2">"ovirtlinode.gsoc.org"</span>
<span class="o">}</span>,
<span class="s2">"changed"</span>: <span class="nb">false</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Running the playbook, assuming you are on <code class="language-plaintext highlighter-rouge">ovirt-ansible/</code> dir</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook site.yml <span class="nt">-i</span> inventory <span class="nt">-vvv</span>
</code></pre></div></div>
<p>This might take a few minutes to get done. Get a coffee or read some xkcd while it does its thing.</p>
<p>You can now access the admin panel through the url <a href="https://ovirtlinode.gsoc.org/ovirt-engine/webadmin/">https://ovirtlinode.gsoc.org/ovirt-engine/webadmin/</a></p>
<p>The url would obviously be different for you, citing the FQDN you decided to use during the installation process.</p>
<p>Cheerio!</p>
Installing oVirt 4.1 on centOS 7 (DigitalOcean)
2017-05-21T00:00:00+00:00
https://www.tasdikrahman.com/2017/05/21/Installing-ovirt-4.1-on-centos-7-digitalocean-Google-Summer-of-Code-oVirt-2017
<p>Was trying to install oVirt engine on a VM deployed on DigitalOcean. My learnings from it are documented here.</p>
<h2 id="installing-ovirt-engine">Installing oVirt Engine</h2>
<p>I would concentrate on the part of just installing oVirt-engine as I had a fair share of problems while doing so.</p>
<p>The VM I am installing it on is a 4GB centOS 7 box with 80GB of SSD to spare for. Also, make sure you read through the whole requirements <a href="http://www.ovirt.org/documentation/quickstart/quickstart-guide/#ovirt-engine">mentioned on the official docs</a> while going forward with this.</p>
<p>A quick run through of what oVirt is. If you have used vSphere by VMWare, this product offered by Redhat is a competitor to it.</p>
<p>The oVirt platform consists of at least one oVirt Engine and one or more Nodes.</p>
<ul>
<li>oVirt Engine provides a graphical user interface to manage the physical and logical resources of the oVirt infrastructure.</li>
<li>oVirt Engine runs virtual machines.</li>
</ul>
<p>oVirt Engine is the control center of the oVirt environment. It allows you to define hosts, configure data centers, add storage, define networks, create virtual machines, manage user permissions and use templates from one central location.</p>
<h2 id="installation">Installation</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]# yum update
<span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]# yum <span class="nb">install </span>http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm
<span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]# yum <span class="nt">-y</span> <span class="nb">install </span>ovirt-engine
<span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]# engine-setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Initializing
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment setup
Configuration files: <span class="o">[</span><span class="s1">'/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf'</span>, <span class="s1">'/etc/ovirt-engine-setup.conf.d/10-packaging.conf'</span><span class="o">]</span>
Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170520124451-wvuuny.log
Version: otopi-1.6.1 <span class="o">(</span>otopi-1.6.1-1.el7.centos<span class="o">)</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment packages setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Programs detection
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment setup
<span class="o">[</span> INFO <span class="o">]</span> Stage: Environment customization
<span class="nt">--</span><span class="o">==</span> PRODUCT OPTIONS <span class="o">==</span><span class="nt">--</span>
Configure Engine on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Configure Image I/O Proxy on this host? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Configure WebSocket Proxy on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Please note: Data Warehouse is required <span class="k">for </span>the engine. If you choose to not configure it on this host, you have to configure it on a remote host, and <span class="k">then </span>configure the engine on this host so that it can access the database of the remote Data Warehouse host.
Configure Data Warehouse on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Configure VM Console Proxy on this host <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
<span class="nt">--</span><span class="o">==</span> PACKAGES <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Checking <span class="k">for </span>product updates...
<span class="o">[</span> INFO <span class="o">]</span> No product updates found
<span class="nt">--</span><span class="o">==</span> NETWORK CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Host fully qualified DNS name of this server <span class="o">[</span>centos-4gb-blr1-ovirt-engine]: ovirt.gsoc.org
<span class="o">[</span>WARNING] Failed to resolve ovirt.gsoc.org using DNS, it can be resolved only locally
Setup can automatically configure the firewall on this system.
Note: automatic configuration of the firewall may overwrite current settings.
Do you want Setup to configure the firewall? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
The following firewall managers were detected on this system: firewalld
Firewall manager to configure <span class="o">(</span>firewalld<span class="o">)</span>:
<span class="o">[</span> ERROR <span class="o">]</span> Invalid value
Firewall manager to configure <span class="o">(</span>firewalld<span class="o">)</span>:
<span class="o">[</span> ERROR <span class="o">]</span> Invalid value
Firewall manager to configure <span class="o">(</span>firewalld<span class="o">)</span>: firewalld
<span class="o">[</span> INFO <span class="o">]</span> firewalld will be configured as firewall manager.
<span class="nt">--</span><span class="o">==</span> DATABASE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Where is the DWH database located? <span class="o">(</span>Local, Remote<span class="o">)</span> <span class="o">[</span>Local]:
Setup can configure the <span class="nb">local </span>postgresql server automatically <span class="k">for </span>the DWH to run. This may conflict with existing applications.
Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? <span class="o">(</span>Automatic, Manual<span class="o">)</span> <span class="o">[</span>Automatic]:
Where is the Engine database located? <span class="o">(</span>Local, Remote<span class="o">)</span> <span class="o">[</span>Local]:
Setup can configure the <span class="nb">local </span>postgresql server automatically <span class="k">for </span>the engine to run. This may conflict with existing applications.
Would you like Setup to automatically configure postgresql and create Engine database, or prefer to perform that manually? <span class="o">(</span>Automatic, Manual<span class="o">)</span> <span class="o">[</span>Automatic]:
<span class="nt">--</span><span class="o">==</span> OVIRT ENGINE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Engine admin password:
Confirm engine admin password:
<span class="o">[</span>WARNING] Password is weak: it is based on a dictionary word
Use weak password? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]: Yes
Application mode <span class="o">(</span>Virt, Gluster, Both<span class="o">)</span> <span class="o">[</span>Both]:
<span class="nt">--</span><span class="o">==</span> STORAGE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Default SAN wipe after delete <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]:
<span class="nt">--</span><span class="o">==</span> PKI CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Organization name <span class="k">for </span>certificate <span class="o">[</span>gsoc.org]:
<span class="nt">--</span><span class="o">==</span> APACHE CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Setup can configure the default page of the web server to present the application home page. This may conflict with existing applications.
Do you wish to <span class="nb">set </span>the application as the default page of the web server? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>Yes]:
Setup can configure apache to use SSL using a certificate issued from the internal CA.
Do you wish Setup to configure that, or prefer to perform that manually? <span class="o">(</span>Automatic, Manual<span class="o">)</span> <span class="o">[</span>Automatic]:
<span class="nt">--</span><span class="o">==</span> SYSTEM CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Configure an NFS share on this server to be used as an ISO Domain? <span class="o">(</span>Yes, No<span class="o">)</span> <span class="o">[</span>No]:
<span class="nt">--</span><span class="o">==</span> MISC CONFIGURATION <span class="o">==</span><span class="nt">--</span>
Please choose Data Warehouse sampling scale:
<span class="o">(</span>1<span class="o">)</span> Basic
<span class="o">(</span>2<span class="o">)</span> Full
<span class="o">(</span>1, 2<span class="o">)[</span>1]:
<span class="nt">--</span><span class="o">==</span> END OF CONFIGURATION <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Setup validation
<span class="o">[</span>WARNING] Cannot validate host name settings, reason: resolved host does not match any of the <span class="nb">local </span>addresses
<span class="o">[</span>WARNING] Less than 16384MB of memory is available
<span class="nt">--</span><span class="o">==</span> CONFIGURATION PREVIEW <span class="o">==</span><span class="nt">--</span>
Application mode : both
Default SAN wipe after delete : False
Firewall manager : firewalld
Update Firewall : True
Host FQDN : ovirt.gsoc.org
Configure <span class="nb">local </span>Engine database : True
Set application as default page : True
Configure Apache SSL : True
Engine database secured connection : False
Engine database user name : engine
Engine database name : engine
Engine database host : localhost
Engine database port : 5432
Engine database host name validation : False
Engine installation : True
PKI organization : gsoc.org
DWH installation : True
DWH database secured connection : False
DWH database host : localhost
DWH database user name : ovirt_engine_history
DWH database name : ovirt_engine_history
DWH database port : 5432
DWH database host name validation : False
Configure <span class="nb">local </span>DWH database : True
Configure Image I/O Proxy : True
Configure VMConsole Proxy : True
Configure WebSocket Proxy : True
Please confirm installation settings <span class="o">(</span>OK, Cancel<span class="o">)</span> <span class="o">[</span>OK]:
<span class="o">[</span> INFO <span class="o">]</span> Stage: Transaction setup
<span class="o">[</span> INFO <span class="o">]</span> Stopping engine service
<span class="o">[</span> INFO <span class="o">]</span> Stopping ovirt-fence-kdump-listener service
<span class="o">[</span> INFO <span class="o">]</span> Stopping dwh service
<span class="o">[</span> INFO <span class="o">]</span> Stopping Image I/O Proxy service
<span class="o">[</span> INFO <span class="o">]</span> Stopping vmconsole-proxy service
<span class="o">[</span> INFO <span class="o">]</span> Stopping websocket-proxy service
<span class="o">[</span> INFO <span class="o">]</span> Stage: Misc configuration
<span class="o">[</span> INFO <span class="o">]</span> Stage: Package installation
<span class="o">[</span> INFO <span class="o">]</span> Stage: Misc configuration
<span class="o">[</span> INFO <span class="o">]</span> Upgrading CA
<span class="o">[</span> INFO <span class="o">]</span> Initializing PostgreSQL
<span class="o">[</span> INFO <span class="o">]</span> Creating PostgreSQL <span class="s1">'engine'</span> database
<span class="o">[</span> INFO <span class="o">]</span> Configuring PostgreSQL
<span class="o">[</span> INFO <span class="o">]</span> Creating PostgreSQL <span class="s1">'ovirt_engine_history'</span> database
<span class="o">[</span> INFO <span class="o">]</span> Configuring PostgreSQL
<span class="o">[</span> INFO <span class="o">]</span> Creating CA
<span class="o">[</span> INFO <span class="o">]</span> Creating/refreshing Engine database schema
<span class="o">[</span> INFO <span class="o">]</span> Creating/refreshing DWH database schema
<span class="o">[</span> INFO <span class="o">]</span> Configuring Image I/O Proxy
<span class="o">[</span> INFO <span class="o">]</span> Setting up ovirt-vmconsole proxy helper PKI artifacts
<span class="o">[</span> INFO <span class="o">]</span> Setting up ovirt-vmconsole SSH PKI artifacts
<span class="o">[</span> INFO <span class="o">]</span> Configuring WebSocket Proxy
<span class="o">[</span> INFO <span class="o">]</span> Creating/refreshing Engine <span class="s1">'internal'</span> domain database schema
<span class="o">[</span> INFO <span class="o">]</span> Generating post <span class="nb">install </span>configuration file <span class="s1">'/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Transaction commit
<span class="o">[</span> INFO <span class="o">]</span> Stage: Closing up
<span class="o">[</span> INFO <span class="o">]</span> Starting engine service
<span class="o">[</span> INFO <span class="o">]</span> Starting dwh service
<span class="o">[</span> INFO <span class="o">]</span> Restarting ovirt-vmconsole proxy service
<span class="nt">--</span><span class="o">==</span> SUMMARY <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Restarting httpd
Please use the user <span class="s1">'admin@internal'</span> and password specified <span class="k">in </span>order to login
Web access is enabled at:
http://ovirt.gsoc.org:80/ovirt-engine
https://ovirt.gsoc.org:443/ovirt-engine
Internal CA 80:BD:83:EA:FB:EB:F8:BD:F5:22:98:F2:90:57:03:92:62:B2:5A:62
SSH fingerprint: ee:43:05:9e:95:cc:7e:bb:6a:6e:aa:28:0d:4d:c1:08
<span class="o">[</span>WARNING] Less than 16384MB of memory is available
<span class="nt">--</span><span class="o">==</span> END OF SUMMARY <span class="o">==</span><span class="nt">--</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170520124451-wvuuny.log
<span class="o">[</span> INFO <span class="o">]</span> Generating answer file <span class="s1">'/var/lib/ovirt-engine/setup/answers/20170520131443-setup.conf'</span>
<span class="o">[</span> INFO <span class="o">]</span> Stage: Pre-termination
<span class="o">[</span> INFO <span class="o">]</span> Stage: Termination
<span class="o">[</span> INFO <span class="o">]</span> Execution of setup completed successfully
<span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]#
</code></pre></div></div>
<p>The most important thing to note above in the <code class="language-plaintext highlighter-rouge">engine-setup</code> command is the FQDN for the VM. You need this for accessing the Engine administration portal.</p>
<p>Take a look at this specific part</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
<span class="o">[</span> INFO <span class="o">]</span> Restarting httpd
Please use the user <span class="s1">'admin@internal'</span> and password specified <span class="k">in </span>order to login
Web access is enabled at:
http://ovirt.gsoc.org:80/ovirt-engine
https://ovirt.gsoc.org:443/ovirt-engine
...
</code></pre></div></div>
<p>For resolving the admin portal, we need to add an entry inside <code class="language-plaintext highlighter-rouge">/etc/hosts</code> inside the VM first.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]# <span class="nb">cat</span> /etc/hosts
<span class="c"># Your system has configured 'manage_etc_hosts' as True.</span>
<span class="c"># As a result, if you wish for changes to this file to persist</span>
<span class="c"># then you will need to either</span>
<span class="c"># a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl</span>
<span class="c"># b.) change or remove the value of 'manage_etc_hosts' in</span>
<span class="c"># /etc/cloud/cloud.cfg or cloud-config from user-data</span>
<span class="c"># The following lines are desirable for IPv4 capable hosts</span>
127.0.0.1 centos-4gb-blr1-ovirt-engine centos-4gb-blr1-ovirt-engine
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4
<span class="c"># The following lines are desirable for IPv6 capable hosts</span>
::1 centos-4gb-blr1-ovirt-engine centos-4gb-blr1-ovirt-engine
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
xxx.xx.xx.xxx ovirt.gsoc.org
<span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]#
</code></pre></div></div>
<p>Where xxx.xx.xx.xxx would be the public IP of my VM.</p>
<p>So if you now do a <code class="language-plaintext highlighter-rouge">ping</code> to <code class="language-plaintext highlighter-rouge">ovirt.gsoc.org</code>, out local DNS would be successfully able to resolve the FQDN</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]# ping ovirt.gsoc.org
PING ovirt.gsoc.org <span class="o">(</span>xxx.xx.xx.xxx<span class="o">)</span> 56<span class="o">(</span>84<span class="o">)</span> bytes of data.
64 bytes from ovirt.gsoc.org <span class="o">(</span>xxx.xx.xx.xxx<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span>1 <span class="nv">ttl</span><span class="o">=</span>64 <span class="nb">time</span><span class="o">=</span>0.034 ms
64 bytes from ovirt.gsoc.org <span class="o">(</span>xxx.xx.xx.xxx<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span>2 <span class="nv">ttl</span><span class="o">=</span>64 <span class="nb">time</span><span class="o">=</span>0.041 ms
64 bytes from ovirt.gsoc.org <span class="o">(</span>xxx.xx.xx.xxx<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span>3 <span class="nv">ttl</span><span class="o">=</span>64 <span class="nb">time</span><span class="o">=</span>0.033 ms
^C
<span class="nt">---</span> ovirt.gsoc.org ping statistics <span class="nt">---</span>
3 packets transmitted, 3 received, 0% packet loss, <span class="nb">time </span>1999ms
rtt min/avg/max/mdev <span class="o">=</span> 0.033/0.036/0.041/0.003 ms
<span class="o">[</span>root@centos-4gb-blr1-ovirt-engine ~]#
</code></pre></div></div>
<p>Now on the local dev box, I did the same thing with my <code class="language-plaintext highlighter-rouge">/etc/hosts</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat</span> /etc/hosts
...
<span class="c"># DO server</span>
xxx.xx.xx.xxx ovirt.gsoc.org
...
</code></pre></div></div>
<p>Now it’s resolvable from my local dev box too</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ping ovirt.gsoc.org
PING ovirt.gsoc.org <span class="o">(</span>xxx.xx.xx.xxx<span class="o">)</span>: 56 data bytes
64 bytes from xxx.xx.xx.xxx: <span class="nv">icmp_seq</span><span class="o">=</span>0 <span class="nv">ttl</span><span class="o">=</span>59 <span class="nb">time</span><span class="o">=</span>5.322 ms
^C
<span class="nt">---</span> ovirt.gsoc.org ping statistics <span class="nt">---</span>
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev <span class="o">=</span> 5.322/5.322/5.322/0.000 ms
</code></pre></div></div>
<h2 id="admin-portal">Admin portal</h2>
<p>Now on your dev box, check the admin panel by going to the url https://ovirt.gsoc.org:443/ovirt-engine</p>
<center><img src="/content/images/2017/05/ovirt-admin-landingpage_840x610.png" /></center>
<p>Let’s take the admin panel for a spin</p>
<center><img src="/content/images/2017/05/ovirt-admin-panel.png" /></center>
<p>Isn’t she pretty?</p>
<p>Had been fighting with some dumb defaults for almost an hour. It’s 5 am. in the morning. So I need to get some sleep. Stay tuned!</p>
<h2 id="debugging-tips">Debugging tips</h2>
<ul>
<li>make sure its (the remote host where you are installing <code class="language-plaintext highlighter-rouge">ovirt-engine</code>) resolvable from your machine.</li>
<li>check if the engine is running (check <code class="language-plaintext highlighter-rouge">/var/log/engine.log</code> and <code class="language-plaintext highlighter-rouge">ovirt-engine</code> service)</li>
<li>check firewall on the machine.</li>
<li>try to connect to the web browser from the virtual machine to be sure its not network issue between you and VM <code class="language-plaintext highlighter-rouge">ping your.vm.com</code> should resolve that ip address</li>
</ul>
<h2 id="links">Links</h2>
<ul>
<li><a href="http://www.ovirt.org/documentation/quickstart/quickstart-guide/">http://www.ovirt.org/documentation/quickstart/quickstart-guide/</a></li>
</ul>
Community bonding period, GSoC 2017 with oVirt org
2017-05-20T00:00:00+00:00
https://www.tasdikrahman.com/2017/05/20/Community-Bonding-period-Google-Summer-of-Code-oVirt-2017
<p>A lot has happened over the last few weeks.</p>
<p>Chelsea won the premiership and that too with a comfortable lead. Dominance is something which we definitely had in the premiership. But I must say, West Brom did put up a good show.</p>
<p>The same day, <a href="https://en.wikipedia.org/wiki/WannaCry_ransomware_attack">WannaCry Ransomware</a> started it’s havoc. If you are affected, be sure to check out <a href="https://github.com/gentilkiwi/wanakiwi">wanawiki</a> and <a href="https://github.com/gentilkiwi/wanadecrypt">wanadecrypt</a>. Benjamin does a wonderful job in explaining the intrinsic details of the tools <a href="https://blog.comae.io/wannacry-decrypting-files-with-wanakiwi-demo-86bafb81112d">in this blog post</a>.</p>
<p><a href="https://techcrunch.com/2017/05/20/btc2k/">Bitcoin just surged past $2,000 for the first time</a>. Sadly I don’t own any.</p>
<p>Also, we are just a week away from the official coding time period for GSoC 2017 which starts on 30th May.</p>
<h2 id="recent-developments-in-the-ovirt-community">Recent developments in the oVirt community</h2>
<p>I have been subscribed to the <a href="http://ovirt.org">oVirt</a> devel mailing lists for quite some time now. Interacting on minor discussions to get a feel of the whole community. #ovirt IRC is something which I haven’t been quite active on.</p>
<p>Some recent developments for our latest 4.2 release lined up. The stable one would be 4.1 for the moment. Check out <a href="http://www.ovirt.org/blog/">oVirt blog</a> for what we have been up to lately.</p>
<p>Our infra team made some changes on our gerrit UI to make it fit better with other oVirt services. Which looks great by the way. Check it out here at <a href="https://gerrit-staging.phx.ovirt.org/">https://gerrit-staging.phx.ovirt.org/</a></p>
<p>Check it out and if any suggestions the new UI, you can create a JIRA ticket here <a href="https://ovirt-jira.atlassian.net/browse/OVIRT-912">https://ovirt-jira.atlassian.net/browse/OVIRT-912</a></p>
<p>Also, if you like living your life on the edge. Check out the nightly builds here <a href="https://www.ovirt.org/develop/dev-process/install-nightly-snapshot/">https://www.ovirt.org/develop/dev-process/install-nightly-snapshot/</a>.</p>
<p>We are also having some enhanced OVA support. Which would be</p>
<ol>
<li>Support for uploading OVA.</li>
<li>Support for exporting a VM/template as OVA.</li>
<li>Support for importing OVA that was generated by oVirt (today, we only support those that are VMware-compatible).</li>
<li>Support for downloading OVA.</li>
</ol>
<p>That’s about it for this time. Happy weekend :)</p>
Making of Trumporate: Building markovipy - Part 1
2017-05-06T00:00:00+00:00
https://www.tasdikrahman.com/2017/05/06/Making-of-trumporate-using-markovipy-generating-sentences-using-markov-chains-part-1
<h2 id="do-you-even-read-comics">Do you even read comics?</h2>
<p>Kiddin. Between, I love reading <a href="http://www.calvinandhobbes.co.uk/">Calvin and Hobbes</a>. It’s something which I keep re-reading their well worn collections, maybe for the n-th time. The thing which keeps me hooked to it maybe the blunt truthfulness of strip.</p>
<p>Haven’t read any?</p>
<p>I like this one because of its simplicity.</p>
<center><img src="/content/images/2017/05/simplicity-calvin.jpg" /></center>
<p>This one makes me smile all the time.</p>
<center><img src="/content/images/2017/05/calvin-smile.jpg" /></center>
<p><a href="https://xkcd.com/">xkcd</a> is the only thing that comes close in comic strips which I visit frequently.</p>
<h2 id="psst-let-me-tell-you-something">Psst. Let me tell you something</h2>
<p>The picture which you saw, titled is “calvin and Markov”, is generated using <a href="https://en.wikipedia.org/wiki/Markov_chain">Markov Chains</a> as explained <a href="http://www.joshmillard.com/2015/07/06/calvin-and-markov/">here by the author</a>. So it’s not written by a human but generated programmitcally using a corpus.</p>
<h2 id="markov-what">Markov What?</h2>
<p>Quite simply put, it theorises that</p>
<blockquote>
<p>every event happening is dependent on its previous event that occurred.</p>
</blockquote>
<p>When the probability of some event is dependent or is conditional on previous events, we say they are dependent events.</p>
<p>Let me put it this way.</p>
<p>You could relate to it from the fact that most of the things in the physical world are dependent on their previous outcomes.</p>
<p>Imagine a coin flip, which is dependent on previous outcomes. So it has short-term memory of one event. This can be visualised using a hypothetical machine which contains two cups, which we call states. In one state we have a 50-50 mix of light versus dark beads, while in the other state we have more dark versus light. One cup we can call state zero.</p>
<p>It represents a dark having previously occurred, and the other state, we can call one, it represents a light bead having previously occurred. To run our machine, we simply start in a random state and make a selection. Then we move to either state zero or one, depending on that event. Based on the outcome of that selection, we output either a zero if it’s dark, or a one if it’s light.</p>
<p>With this two-state machine, we can identify four possible transitions. If we are in state zero and a black occurs, we loop back to the same state and select again. If a light bead is selected, we jump over to state one, which can also loop back on itself, or jump back to state zero if a dark is chosen. The probability of a light versus dark selection is clearly not independent here, since it depends on the previous outcome.</p>
<p>But Markov proved that as long as every state in the machine is reachable, when you run these machines in a sequence, they reach equilibrium. That is, no matter where you start, once you begin the sequence, the number of times you visit each state converges to some specific ratio, or a probability.</p>
<p>Quite naturally , if it rains when it’s a cloudy day. You don’t drown when you are in your bed.</p>
<p>This helps in calculating the conditional probability and can be applied to various scenarios.</p>
<center><img src="/content/images/2017/05/markovdiag.png" /></center>
<h2 id="why-am-i-so-interested-in-it">Why am I so interested in it?</h2>
<p>Maybe because it comes into the intersection of math and linguistics. Or maybe I wanted something new to fool around. Take your pick. You wouldn’t be wrong both ways.</p>
<p>Going down the rabbit hole I wanted to build something of my own with this new found knowledge from the read ups.</p>
<center><img src="/content/images/2017/05/markovipy.png" /></center>
<p>I built Markovipy, a Markov Text Generator which can be used to randomly generate (somewhat) realistic sentences, using words from a source text. Words are joined together in sequence, with each new word being selected based on how often it follows the previous word in the source document.</p>
<p>The results are often just nonsense, but at times can be strangely poetic - the sentences below were generated from the text of “The Hamlet” by Shakespeare:</p>
<blockquote>
<p>If his occulted guilt, Do not it selfe vnkennell in one speech, It is most retrograde to our desire And we beseech you, bend you to remaine Heere in the cheere and comfort of our eye, Our cheefest Courtier Cosin, and our whole Kingdome To be contracted in one brow of woe Yet so farre hath Discretion fought with Nature, That we with wisest sorrow thinke on him, Together with remembrance of our selues.</p>
</blockquote>
<p>Here I was using a chain length of 3 which optionally represents the number of words taken into account when choosing the next word. Chain length defaults to 1 (which is fastest), but increasing this may generate more realistic text, albeit slightly more slowly.</p>
<p>Depending on the text, increasing the chain length past 6 or 7 words probably won’t do much good – at that point you’re usually plucking out whole sentences anyway, so using a Markov model is kind of redundant.</p>
<p>I got them off Project Gutenberg. The usual copyright headers had to be removed so that they could serve as useful sample input, but naturally all the rights and restrictions of a Gutenberg book still apply.</p>
<p>The API looks something like this</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span>
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">markovipy.markovipy</span> <span class="kn">import</span> <span class="n">MarkoviPy</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MarkoviPy</span><span class="p">(</span><span class="s">"/Users/tasrahma/development/projects/markovipy/corpus/shakespeare/hamlet_utf8.txt"</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">generate_sentence</span><span class="p">()</span>
<span class="s">'If his occulted guilt, Do not it selfe vnkennell in one speech, It is most retrograde to our desire And we beseech you, bend you to remaine Heere in the cheere and comfort of our eye, Our cheefest Courtier Cosin, and our whole Kingdome To be contracted in one brow of woe Yet so farre hath Discretion fought with Nature, That we with wisest sorrow thinke on him, Together with remembrance of our selues.'</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">generate_sentence</span><span class="p">()</span>
<span class="s">'Fare you well my Lord Ham.'</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">generate_sentence</span><span class="p">()</span>
<span class="s">'To thinke, my Lord? Ham.'</span>
</code></pre></div></div>
<h2 id="future-improvements">Future improvements</h2>
<p>As with every project, there is always space to improve and here are some things which can be done with markovipy</p>
<ul>
<li>Specify the number of sentences to be generated when the API is being called. As of now only one sentence gets generated till the period.</li>
<li>I am storing the mappings of possible words in memory as of now which can be shifted to redis</li>
<li>Or if you can suggest something, I would be very happy to take a look. Create an issue here on it’s github page. <a href="https://nlp.stanford.edu/IR-book/html/htmledition/markov-chains-1.html">https://github.com/tasdikrahman/markovipy/issues/</a></li>
</ul>
<p>I would like to end this article using the last issue of Calvin and Hobbes which was on 31st December, 1995. The same year and month that I was born :)</p>
<center><img src="/content/images/2017/05/calvin-hobbes-final-strip-dec-31-1995.jpg" /></center>
<p>Thanks for your time. Cheers!</p>
<p><strong>DISCLAIMER: All Calvin and Hobbes Comic strips and images copyright to Bill Watterson and publishers. All written © CalvinAndHobbes.co.uk</strong></p>
<h2 id="further-read">Further read</h2>
<ul>
<li><a href="https://github.com/tasdikrahman/markovipy/">https://github.com/tasdikrahman/markovipy/</a></li>
<li><a href="http://www.joshmillard.com/2015/07/06/calvin-and-markov/">http://www.joshmillard.com/2015/07/06/calvin-and-markov/</a></li>
<li><a href="http://setosa.io/ev/markov-chains/">http://setosa.io/ev/markov-chains/</a></li>
<li><a href="https://nlp.stanford.edu/IR-book/html/htmledition/markov-chains-1.html">https://nlp.stanford.edu/IR-book/html/htmledition/markov-chains-1.html</a></li>
</ul>
Hello oVirt, GSoC 2017
2017-05-04T00:00:00+00:00
https://www.tasdikrahman.com/2017/05/04/Got-accepted-for-Google-Summer-of-Code-oVirt-2017
<p>So Chelsea won their last Premier league match against Everton hands down last Sunday. And we had a very comfortable win I would say.</p>
<p>Pedro’s 25-yard stunner, Gary Cahill’s close-range finish and Willian’s tap-in kept us ahead of The toffees. Courtois got his much deserved clean sheet. Wouldn’t be wrong to say that we had a great day.</p>
<p>We are just 3 wins away from the premier league title with Tottenham right behind our backs.</p>
<p>What other better news can there be for me?</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Accepted for <a href="https://twitter.com/gsoc">@gsoc</a> 2017! Will write software for <a href="https://twitter.com/ovirt">@ovirt</a> under <a href="https://twitter.com/redhatopen">@redhatopen</a>. Couldn't be any happier 😄 <a href="https://twitter.com/hashtag/GSOC?src=hash">#GSOC</a> <a href="https://twitter.com/hashtag/gsoc2017?src=hash">#gsoc2017</a> <a href="https://twitter.com/hashtag/RedHat?src=hash">#RedHat</a> <a href="https://twitter.com/hashtag/opensource?src=hash">#opensource</a></p>— Tasdik Rahman (@tasdikrahman) <a href="https://twitter.com/tasdikrahman/status/860188737376067587">May 4, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="google-summer-of-code">Google Summer of Code</h2>
<p>Well turns out there is. I got accepted for GSoC 2017 under <a href="http://ovirt.org">oVirt</a> and will be working with <a href="https://cz.linkedin.com/in/lukas-svaty-4236b573">Lukas Svaty</a> among others this summer.</p>
<blockquote>
<p>Building things which ease human effort is something which I deeply believe in.</p>
</blockquote>
<p>I used to read a lot of <a href="https://xkcd.com/">xkcd</a> so ended up making an <a href="https://github.com/tasdikrahman/xkcd-dl">xkcd-dl</a>, a cross platform comic downloader for readers. I liked playing classic FPS games so I ended up writing <a href="https://github.com/tasdikrahman/spaceShooter">spaceShooter</a>, a cross platform game using pygame which ended up on HN, Product hunt and what not. Used to spend most of the time at the terminal, had a lot of <code class="language-plaintext highlighter-rouge">.txt</code> files lying around which had random links or pieces of text conjured from couch surfing. Thought I should keep them at a unified place with tags and a searcheable interface. Hacked together <a href="https://github.com/tasdikrahman/tnote">tnote</a> which does just that. And so on for my other projects.</p>
<blockquote>
<p>There was a need and hence I built it!</p>
</blockquote>
<p>GSoC is all about coming together with the community and building things which would be used by thousands(if not millions). And oVirt would allow me to live by that very statement.</p>
<p>oVirt is a product very similar to <a href="http://www.vmware.com/in/products/vsphere.html">vsphere</a> offered by <a href="http://www.vmware.com/">VMWare</a>. Both of them are competitor of sorts in that space. Both being server virtualisation tools.</p>
<p>The open source project is the downstream version of the commercial product. Just as is the case with other Redhat projects. You have <a href="https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux">RHEL</a> which is the upstream version of <a href="https://www.centos.org/">CentOS</a> which is nothing but the hardened version of <a href="https://getfedora.org/">fedora</a>.</p>
<h2 id="about-my-project-specifically">About my project specifically</h2>
<p>So I have been fooling around a configuration management tool called <a href="https://www.ansible.com/">Ansible</a> for some months now (which would be evident from my feed if you stumble upon my blog once in while). Apparantly, my interests co-incided with one of the project proposals at oVirt. So there you go.</p>
<p>If you are curious about the whole project proposal, here is <a href="(https://summerofcode.withgoogle.com/projects/#6188534896525312)">the link to it</a> for you to read more about it :)</p>
<p>All in all I cannot be more excited about this!</p>
<p>Cheerio!</p>
Testing your ansible roles using travis-CI
2017-04-06T00:00:00+00:00
https://www.tasdikrahman.com/2017/04/06/Testing-your-ansible-roles-using-travis-CI
<p><strong>NOTE</strong>: <strong>The ansible playbook written here can be found at <a href="https://github.com/tasdikrahman/ansible-bootstrap-server">tasdikrahman/ansible-bootstrap-server</a></strong></p>
<h2 id="continous-integration">Continous Integration</h2>
<p>Simply put with each commit that you are making to shared repository, which is then verified by an automated build. This helps in detection of errors early on.</p>
<p>If you are new to this development style. There are <a href="https://martinfowler.com/articles/continuousIntegration.html">plenty</a> <a href="http://softwareengineering.stackexchange.com/questions/198471/simple-explanation-of-continuous-integration">of</a> <a href="https://www.thoughtworks.com/continuous-integration">places</a> which explain will help you understand. This practice in itself is quite old. CI/CD anyone?</p>
<p>But I am not writing this to explain what is CI right?</p>
<h2 id="so-you-made-an-ansible-playbook">So you made an ansible playbook?</h2>
<p>I have talked about Infra as code in some of my earlier blog posts. Automatically provisioning your complete server(s) in minutes is something which every org is trying/has achieved.</p>
<p>If you follow TDD principles, you would be knowing right where I am taking this conversation to.</p>
<p>Here is the directory structure for the ansible role I am testing this out</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-bootstrap-server
├── .travis.yml
├── ansible.cfg
├── play.yml
├── roles
│ ├── basic_server_hardening
│ │ ├── defaults
│ │ │ └── main.yml
│ │ ├── handlers
│ │ │ └── main.yml
│ │ └── tasks
│ │ └── main.yml
│ ├── create_new_user
│ │ ├── defaults
│ │ │ └── main.yml
│ │ └── tasks
│ │ └── main.yml
│ ├── install_minimal_packages
│ │ └── tasks
│ │ └── main.yml
│ ├── update
│ │ └── tasks
│ │ └── main.yml
│ └── vimserver
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ │ └── vimrc_server
│ └── tasks
│ └── main.yml
└── tests
├── inventory
└── test.yml
</code></pre></div></div>
<p><strong>If you want to understand how the files are organised. I have written about it in <a href="http://www.tasdikrahman.com/2017/03/19/Organising-tasks-in-roles-using-Ansible/">“Organising tasks in roles using Ansible”</a></strong></p>
<h2 id="writings-tests">Writings tests</h2>
<p>I would be running the tests inside the travis build environment.</p>
<p>Why travis?</p>
<p>I like them more! But there are many other good CI/CD providers namely circleCI, bambooCI and some more. Choose whatever fits best to your organisation or personal appeal.</p>
<p>So unlike when I am running the <code class="language-plaintext highlighter-rouge">ansible-playbook</code> from the controller node, I would be running the playbook on the <code class="language-plaintext highlighter-rouge">localhost</code>. This part would be obvious by now.</p>
<p>Let’s have a look at <code class="language-plaintext highlighter-rouge">tests/test.yml</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">---</span>
- hosts: localhost
connection: <span class="nb">local
</span>become: <span class="nb">true
</span>roles:
- <span class="o">{</span>role: ../roles/update<span class="o">}</span>
- <span class="o">{</span>role: ../roles/install_minimal_packages<span class="o">}</span>
- <span class="o">{</span>role: ../roles/create_new_user<span class="o">}</span>
- <span class="o">{</span>role: ../roles/basic_server_hardening<span class="o">}</span>
- <span class="o">{</span>role: ../roles/vimserver<span class="o">}</span>
</code></pre></div></div>
<p>Let’s break it down,</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">hosts: localhost</code> this is simply telling the host/group of hosts which this playbook would be targeting.</li>
<li><code class="language-plaintext highlighter-rouge">connection: local</code> would tell ansible to run the tasks on the system itself and not <code class="language-plaintext highlighter-rouge">ssh</code> onto to some remote machine for executing the playbook.</li>
<li><code class="language-plaintext highlighter-rouge">become: true</code> makes the tasks run as the <code class="language-plaintext highlighter-rouge">root</code> user</li>
</ul>
<p>and the <code class="language-plaintext highlighter-rouge">roles</code> part is where we would be organising our roles to be executed sequentially.</p>
<h2 id="testing-against-different-versions-of-ansible">Testing against different versions of Ansible</h2>
<p>For travis to build your code, it would be requiring a <code class="language-plaintext highlighter-rouge">.travis.yml</code> file in the root directory of your project.</p>
<p>In this particualr example, the contents of it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">---</span>
<span class="nb">sudo</span>: required
dist: trusty
language: python
python: <span class="s2">"2.7"</span>
<span class="c"># Doc: https://docs.travis-ci.com/user/customizing-the-build#Build-Matrix</span>
<span class="nb">env</span>:
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>latest
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.2.2.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.2.1.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.2.0.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.1.5
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.1.4
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.1.3
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.1.2
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.1.1.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.1.0.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.0.2.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.0.1.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.0.0.2
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.0.0.1
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>2.0.0.0
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>1.9.6
branches:
only:
- master
before_install:
- <span class="nb">sudo </span>apt-get update <span class="nt">-qq</span>
<span class="nb">install</span>:
<span class="c"># Install Ansible.</span>
- <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$ANSIBLE_VERSION</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"latest"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span>pip <span class="nb">install </span>ansible<span class="p">;</span> <span class="k">else </span>pip <span class="nb">install </span><span class="nv">ansible</span><span class="o">==</span><span class="nv">$ANSIBLE_VERSION</span><span class="p">;</span> <span class="k">fi</span>
- <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$ANSIBLE_VERSION</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"latest"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span>pip <span class="nb">install </span>ansible-lint<span class="p">;</span> <span class="k">fi
</span>script:
<span class="c"># Check the role/playbook's syntax.</span>
- ansible-playbook <span class="nt">-i</span> tests/inventory tests/test.yml <span class="nt">--syntax-check</span>
<span class="c"># Run the role/playbook with ansible-playbook.</span>
- ansible-playbook <span class="nt">-i</span> tests/inventory tests/test.yml <span class="nt">-vvvv</span> <span class="nt">--skip-tags</span> update,copy_host_ssh_id
<span class="c"># check is the user is created or not</span>
- <span class="nb">id</span> <span class="nt">-u</span> tasdik | <span class="nb">grep</span> <span class="nt">-q</span> <span class="s2">"no"</span> <span class="o">&&</span> <span class="o">(</span><span class="nb">echo</span> <span class="s2">"user not found"</span> <span class="o">&&</span> <span class="nb">exit </span>1<span class="o">)</span> <span class="o">||</span> <span class="o">(</span><span class="nb">echo</span> <span class="s2">"user found"</span> <span class="o">&&</span> <span class="nb">exit </span>0<span class="o">)</span>
</code></pre></div></div>
<p>The interesting thing to note here is the list arguments for <code class="language-plaintext highlighter-rouge">env</code> here. Travis expands on these environment variables to create multiple build environments one after another.</p>
<p>For this case we have 16 <code class="language-plaintext highlighter-rouge">env</code> variables, so there would be 16 seperate builds for the specified ansible versions which this playbook will be tested against.</p>
<p>So for the line</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">.</span>
<span class="nb">env</span>:
- <span class="nv">ANSIBLE_VERSION</span><span class="o">=</span>latest
<span class="nb">.</span>
</code></pre></div></div>
<p>The Build config will be something like, where you can see the <code class="language-plaintext highlighter-rouge">"env": "ANSIBLE_VERSION=latest"</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">{</span>
<span class="s2">"sudo"</span>: <span class="s2">"required"</span>,
<span class="s2">"dist"</span>: <span class="s2">"trusty"</span>,
<span class="s2">"language"</span>: <span class="s2">"python"</span>,
<span class="s2">"python"</span>: <span class="s2">"2.7"</span>,
<span class="s2">"env"</span>: <span class="s2">"ANSIBLE_VERSION=latest"</span>,
<span class="s2">"before_install"</span>: <span class="o">[</span>
<span class="s2">"sudo apt-get update -qq"</span>
<span class="o">]</span>,
<span class="s2">"install"</span>: <span class="o">[</span>
<span class="s2">"if [ </span><span class="se">\"</span><span class="nv">$ANSIBLE_VERSION</span><span class="se">\"</span><span class="s2"> = </span><span class="se">\"</span><span class="s2">latest</span><span class="se">\"</span><span class="s2"> ]; then pip install ansible; else pip install ansible==</span><span class="nv">$ANSIBLE_VERSION</span><span class="s2">; fi"</span>,
<span class="s2">"if [ </span><span class="se">\"</span><span class="nv">$ANSIBLE_VERSION</span><span class="se">\"</span><span class="s2"> = </span><span class="se">\"</span><span class="s2">latest</span><span class="se">\"</span><span class="s2"> ]; then pip install ansible-lint; fi"</span>
<span class="o">]</span>,
<span class="s2">"script"</span>: <span class="o">[</span>
<span class="s2">"ansible-playbook -i tests/inventory tests/test.yml --syntax-check"</span>,
<span class="s2">"ansible-playbook -i tests/inventory tests/test.yml -vvvv --skip-tags update,copy_host_ssh_id"</span>,
<span class="s2">"id -u tasdik | grep -q </span><span class="se">\"</span><span class="s2">no</span><span class="se">\"</span><span class="s2"> && (echo </span><span class="se">\"</span><span class="s2">user not found</span><span class="se">\"</span><span class="s2"> && exit 1) || (echo </span><span class="se">\"</span><span class="s2">user found</span><span class="se">\"</span><span class="s2"> && exit 0)"</span>
<span class="o">]</span>,
<span class="s2">"group"</span>: <span class="s2">"stable"</span>,
<span class="s2">"os"</span>: <span class="s2">"linux"</span>
<span class="o">}</span>
</code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">ansible-playbook -i tests/inventory tests/test.yml --syntax-check</code>:</li>
</ul>
<p>would check for any syntax errors as obvious from the command itself, helps in checking any errors early on before the build.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">ansible-playbook -i tests/inventory tests/test.yml -vvvv --skip-tags update,copy_host_ssh_id</code>:</li>
</ul>
<p>is the line which actually runs the playbook</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">id -u tasdik | grep -q "no" && (echo "user not found" && exit 1) || (echo "user found" && exit 0)</code>:</li>
</ul>
<p>checks whether a user named <code class="language-plaintext highlighter-rouge">tasdik</code> exists or not after the playbook has completed its execution.</p>
<p>I have skipped explaining some of the parts in my <code class="language-plaintext highlighter-rouge">.travis.yml</code>. You can learn more about <a href="https://docs.travis-ci.com/user/customizing-the-build">build configuration inside travis from the docs</a></p>
<h2 id="skipping-tasks-inside-build">Skipping tasks inside build</h2>
<p>Travis builds will fail, if you try to exceed a particular limit while your build job is running. You can find more about the exact specs and timings in the <a href="https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts">travis docs talking about build timeouts</a></p>
<p>Some roles in particular had tasks which would make some network calls which is unnecessary to test in a travis build. I needed to skip them in the builds</p>
<p>Enter ansible <code class="language-plaintext highlighter-rouge">tags</code>.</p>
<p>It provides an elegant way to skip or only run tasks with some specified tags.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">.</span>
<span class="s2">"ansible-playbook -i tests/inventory tests/test.yml -vvvv --skip-tags update,copy_host_ssh_id"</span>
<span class="nb">.</span>
</code></pre></div></div>
<p>The above playbook run will run the playbook <code class="language-plaintext highlighter-rouge">tests/test.yml</code> on the hosts described on <code class="language-plaintext highlighter-rouge">tests/inventory</code> skipping the tags <code class="language-plaintext highlighter-rouge">update,copy_host_ssh_id</code> which I have specified inside my tasks.</p>
<p>Would be trying out testing ansible roles on docker next. Stay tuned.</p>
<p>Happy ansibling!</p>
<p><strong>Further reading</strong></p>
<ul>
<li><a href="http://docs.ansible.com/ansible/playbooks_tags.html">http://docs.ansible.com/ansible/playbooks_tags.html</a></li>
<li><a href="https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts">https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts</a></li>
<li><a href="https://docs.travis-ci.com/user/customizing-the-build">https://docs.travis-ci.com/user/customizing-the-build</a></li>
<li><a href="https://www.jeffgeerling.com/blog/testing-ansible-roles-travis-ci-github">https://www.jeffgeerling.com/blog/testing-ansible-roles-travis-ci-github</a></li>
</ul>
Organising tasks in roles using Ansible
2017-03-19T00:00:00+00:00
https://www.tasdikrahman.com/2017/03/19/Organising-tasks-in-roles-using-Ansible
<p><strong>NOTE</strong>: <strong>The ansible playbook written here can be found at <a href="https://github.com/tasdikrahman/ansible-playbooks/tree/master/digitalocean">tasdikrahman/ansible-playbook</a></strong></p>
<p>Roles are nothing but a further abstraction of making your playbook more modular. If you have played around with the <code class="language-plaintext highlighter-rouge">ansible-playbook</code> command. You might have noticed the common pattern of repeating tasks which you did some or the other time back.</p>
<p>Ansible roles provide you a way to reuse tasks(or roles for that matter). Imagine this to be a very similar concept writing Object oriented code.</p>
<h2 id="need-for-roles">Need for roles?</h2>
<p>I realised that I was doing the same thing over and over again whenever I had to spin up a new droplet(instance for the EC2 people). Things like</p>
<ul>
<li>Updating and upgrading the existing packages</li>
<li>Installing some bare essentials on it (eg: git, vim, ncdu etc)</li>
<li>creating a non-root user with admin privileges</li>
<li>enabling a basic firewall</li>
<li>some common chores.</li>
</ul>
<p>Hence I found myself writing <a href="https://github.com/tasdikrahman/ansible-playbooks/tree/master/digitalocean">tasdikrahman/ansible-playbook</a></p>
<p>Take this structure for example.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>tree digitalocean
digitalocean
├── README.md
├── play.yml
└── roles
├── bootstrap_server
│ └── tasks
│ └── main.yml
├── create_new_user
│ └── tasks
│ └── main.yml
├── update
│ └── tasks
│ └── main.yml
└── vimserver
├── files
│ └── vimrc_server
└── tasks
└── main.yml
</code></pre></div></div>
<p>Let’s break it down further.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">digitalocean</code> : The root dir which will contain the <code class="language-plaintext highlighter-rouge">roles</code> dir which further contains roles for tasks</li>
<li><code class="language-plaintext highlighter-rouge">play.yml</code> : A normal playbook in <code class="language-plaintext highlighter-rouge">.yml</code> format which stiches the roles that we have created inside <code class="language-plaintext highlighter-rouge">roles</code> dir</li>
<li><code class="language-plaintext highlighter-rouge">bootstrap_server</code> : contains the task file(s) needed for the necessary tasks declared inside the <code class="language-plaintext highlighter-rouge">tasks/main.yml</code>. The roles <code class="language-plaintext highlighter-rouge">create_new_user</code> et el suggest the same thing.</li>
<li><code class="language-plaintext highlighter-rouge">tasks</code> : This directory contains all of the tasks that would normally be in a playbook. These can reference files and templates contained in their respective directories without using a path.</li>
</ul>
<p>Similar to <code class="language-plaintext highlighter-rouge">tasks</code> dir inside roles, we have many more files specifically used for other things</p>
<p>Those being,</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">files</code>: This directory contains regular files that need to be transferred to the hosts you are configuring for this role. This may also include script files to run.</li>
<li><code class="language-plaintext highlighter-rouge">handlers</code>: All handlers that were in your playbook previously can now be added into this directory.</li>
<li><code class="language-plaintext highlighter-rouge">meta</code> : This directory can contain files that establish role dependencies. You can list roles that must be applied before the current role can work correctly.</li>
<li><code class="language-plaintext highlighter-rouge">templates</code>: You can place all files that use variables to substitute information during creation in this directory.</li>
<li><code class="language-plaintext highlighter-rouge">vars</code>: Variables for the roles can be specified in this directory and used in your configuration files.</li>
</ul>
<h2 id="playyml"><code class="language-plaintext highlighter-rouge">play.yml</code></h2>
<p>The contents of my <code class="language-plaintext highlighter-rouge">play.yml</code> are sequenced in such a manner so that the roles get executed in a sequence. This is a very handy feature which allows you to configure software which depends on some other software.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
- hosts: testdroplets
roles:
- update
- bootstrap_server
- role: create_new_user
username: tasdik
- role: vimserver
username: tasdik
</code></pre></div></div>
<p>The roles are placed as key-value’s inside the dict roles here. the <code class="language-plaintext highlighter-rouge">username</code> variable is being passed to the roles <code class="language-plaintext highlighter-rouge">create_new_server</code> and <code class="language-plaintext highlighter-rouge">vimserver</code> as a value using which the new user should be created.</p>
<p>You can also put the <code class="language-plaintext highlighter-rouge">username</code> variable inside individual roles dir, which would look something like</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>└── roles
└── create_new_user
├── vars
│ └── main.yml
└── tasks
└── main.yml
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">vars/main.yml</code> would contain</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
username: tasdik
</code></pre></div></div>
<p>But I feel this would become a cumbersome task for the ansible-playbook at question. I would again have to put the same <code class="language-plaintext highlighter-rouge">vars/main.yml</code> inside the role <code class="language-plaintext highlighter-rouge">vimserver</code> which would be a duplicate of this file.</p>
<p>For that reason, I am passing the variable at the play level for each of the ansible roles.</p>
<h2 id="how-does-jinja2-come-into-play-here">How does Jinja2 come into play here?</h2>
<p>Have a look at the file</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>digitalocean/roles/vimserver/tasks/main.yml
<span class="nt">---</span>
- name: Place vimrc_server on ~/.vimrc
copy:
src: vimrc_server
dest: /home/<span class="o">{</span> username <span class="o">}</span>/.vimrc
mode: 0644
owner: <span class="s2">"{ username }"</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">"{ username }"</code> is substitued with the value that you provided to the role at runtime. The reason we have put quotes around it is because the yaml parser for ansible requires it so when you are just starting the jinja variable as the starting value.</p>
<p>You needn’t have put the quotes if for example it were something like this.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- name: Copy .ssh/id_rsa from host box to the remote box <span class="k">for </span>user username
become: <span class="nb">true
</span>copy:
src: ~/.ssh/id_rsa.pub
dest: /home/<span class="o">{</span> username <span class="o">}</span>/.ssh/authorized_keys
</code></pre></div></div>
<p><strong>NOTE: Please note that there are double curly braces in the above example which surround <code class="language-plaintext highlighter-rouge">username</code>. The templating engine wasn’t allowing me to put it there as it would take it as a variable for substitution.</strong></p>
<h2 id="closing-thoughts">Closing thoughts</h2>
<p>Ansible uses Jinja2 as the templating engine of choice(also the choice of the biggies Flask and Django). So you would be in familair territory if you have dabbled in those.</p>
<p>You can finally run this playbook using</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>ansible-playbook play.yml <span class="nt">-vvv</span>
</code></pre></div></div>
<p>Happy ansibling!</p>
Introduction to Configuration Management using Ansible
2017-02-28T00:00:00+00:00
https://www.tasdikrahman.com/2017/02/28/Ansible-Introduction-to-Configuration-Management
<h2 id="need-for-configuration-management">Need for Configuration management</h2>
<p>There are many devs/sysadmins out there who manage their servers by logging in through ssh. Making the changes and then logging out again. Sounds like you?</p>
<p>Well hey, you are not alone!</p>
<p>But do you feel that this can create snowflake servers?</p>
<p>Servers which are impossible to recreate because we missed out on some minute detail which the other dev had known.</p>
<blockquote>
<p>But Tasdik. This wouldn’t happen if we have a very good documentation process giving a step by step guide on how to do so!</p>
</blockquote>
<p>Good! I will say you guys have followed a very good engineering practice of documenting each and every other process that you do! But in most fast moving
environments. This is not the case!</p>
<h2 id="enter-configuration-management">Enter Configuration management</h2>
<p>It’s good that we have a good range of config. management tools out there like <a href="http://cfengine.com/">CFEngine</a>, <a href="http://ansible.com/">Ansible</a>, <a href="http://getchef.com/chef">chef</a> to name a few.</p>
<h2 id="isnt-this-devops-thing-some-buzzword-out-there">Isn’t this DevOps thing some buzzword out there?</h2>
<p>Let me tell you a first hand experience of mine which really made me think about provisioning tools.</p>
<p>I am currently working on a project which deals with <code class="language-plaintext highlighter-rouge">OpenCV</code> 3 and some python dependencies thrown around. We have currently deployed the demo app on a humble 512MB RAM, 20 gig SSD on a DigitalOcean droplet which runs Xenial Xerius(64 bit).</p>
<p>The general process of getting the development env up and running for OpenCV involves</p>
<ul>
<li>Updating and upgrading the dist packages</li>
<li>installing developer toolchains for compiling and installing the <code class="language-plaintext highlighter-rouge">C++</code> source code</li>
<li>installing tons of libraries for image and video formats</li>
<li>getting the actual source code of OpenCV and OpenCV contrib which we are gonna use</li>
<li>Build a particular version of OpenCV required by us.</li>
<li>Compiling it</li>
</ul>
<p>And the list goes on!</p>
<h2 id="but-you-can-still-write-a-shell-script-for-it-right">But you can still write a shell script for it. Right?</h2>
<p>Yes! You absolutely can. No doubt.</p>
<p>I will go further one step and say that it DOES take time and resources to learn a config management tool and have some working knowlege to be productive with it.</p>
<p>But here are some points in favor for config mgmt. tools</p>
<ul>
<li>They can be re-used, distributed. Chef has cookbooks, juju has charms, Perl has CPAN, Java has Maven.</li>
<li>The DSL’s do take away some freedom, but they are much cleaner.</li>
<li><strong>Idempotency</strong>: Which means you can safely re-run it any number of times and each time it will go to the desired state, remain there or more closer to the desired state.</li>
<li>Scalable!</li>
<li>OS Agnostic.</li>
<li>Version Controlling - In short, maintaining Infrastructure as Code.</li>
<li>Easy to write</li>
</ul>
<p>This is an example <code class="language-plaintext highlighter-rouge">ansible</code> role which updates the <code class="language-plaintext highlighter-rouge">apt-cache</code> for target server.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
- name: Updating the apt-cache
become: yes
apt: update_cache=true
</code></pre></div></div>
<p>Looks good?</p>
<h2 id="why-ansible">Why Ansible?</h2>
<p>I just wanted to get started with some or the other tool! That’s it.</p>
<p>I am pretty sure that <code class="language-plaintext highlighter-rouge">chef</code>, <code class="language-plaintext highlighter-rouge">puppet</code>, <code class="language-plaintext highlighter-rouge">salt stack</code> and the like are equally good and will serve your use case. So do check them out too and have a feel around the different hammers around the market.</p>
<p>But one great thing about Ansible is that you don’t need a PKI architecture or some special communication protocol for managing it’s nodes. It(your manager) just uses plain on SSH for communicating with it’s nodes. Although for older versions of ansible, it used to communicate using the paramiko SSH-2 python implementation</p>
<h2 id="a-simple-comparison">A simple comparison</h2>
<p>Nothing much, just a simple <code class="language-plaintext highlighter-rouge">apache2</code> virtual hosts setup for your VPS.</p>
<h4 id="manual-install">Manual install</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo apt-get update && upgrade
$ sudo apt-get install apache2
$ cd /etc/apache2
/etc/apache2 $ sudo cp /files/awesome-app sites-available/awesome-app.conf
/etc/apache2 $ sudo chmod 640 sites-available/awesome-app.conf
/etc/apache2 $ sudo rm /etc/apache2/sites-enabled/default && /etc/apache2/sites-enabled/default-ssl
/etc/apache2 $ sudo ln -s /etc/apache2/sites-available/awesome-app /etc/apache2/sites-enabled/awesome-app
/etc/apache2 $ sudo service apache2 restart
</code></pre></div></div>
<h4 id="shell-script">Shell script</h4>
<p>You can put all the necessary commands above and put it inside a <code class="language-plaintext highlighter-rouge">provision.sh</code> file and then call it using <code class="language-plaintext highlighter-rouge">$ sh provision.sh</code></p>
<h4 id="ansible-playbook">Ansible playbook</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- hosts: web
become: yes # for escalated privileges
tasks:
- name: Installs apache web server
apt: pkg=apache2 state=installed update_cache=true
- name: Push default virtual host configuration
copy: src=files/awesome-app dest=/etc/apache2/sites-available/awesome-app mode=0640
- name: Disable the default virtualhost
file: dest=/etc/apache2/sites-enabled/default state=absent
notify:
- restart apache
- name: Disable the default ssl virtualhost
file: dest=/etc/apache2/sites-enabled/default-ssl state=absent
notify:
- restart apache
- name: Activates our virtualhost
file: src=/etc/apache2/sites-available/awesome-app dest=/etc/apache2/sites-enabled/awesome-app state=link
notify:
- restart apache
handlers:
- name: restart apache
service: name=apache2 state=restarted
</code></pre></div></div>
<p>The above configuration is just a simple POC showing the general relative ease of using tools like <code class="language-plaintext highlighter-rouge">Ansible</code>. You can do a lot more like provisioning full blown distributed servers from scratch, load balance them by putting a load balancer like HAProxy in front of it and what not!</p>
<p>So it’s left on you to decide what suits you best.</p>
<p>Cheers!</p>
<p>On a side note, I automated installing and setting up the dev environment for OpenCV 3 for python3 using Ansible. The code as usual for every other project lies in <a href="https://github.com/tasdikrahman/opencv3-ansible-vagrant-playbook">github</a></p>
More than 18 stops, a little less than 1800kms, Backpacking Trip To Himachal Pradesh
2016-12-22T00:00:00+00:00
https://www.tasdikrahman.com/2016/12/22/solo-backpacking-trip-himachal-pradesh-shimla-manali-kullu-kasol-manikaran-bhuntar-triund-dharamshala-mcleodganj-budget-india
<h2 id="trip-itinerary">Trip itinerary</h2>
<p>We all have that one trip with friends which gets cancelled no matter what right(goa anybody!)?</p>
<iframe src="https://uploads.knightlab.com/storymapjs/f8903d09026b4be3c6e94eca06c8e73e/himachal-trip/index.html" frameborder="0" width="100%" height="800"></iframe>
<p>But hopefully, our decided trip was completed. Wasn’t that smooth. But as the saying goes</p>
<blockquote>
<p>What is an ocean if it doesn’t challenge the sailors!</p>
</blockquote>
<p>Thanks to the recent <a href="http://www.wsj.com/articles/indias-demonetization-debacle-1481851086">demonetization drive</a> by the Indian govt. We had to delay our plans for the trip by some days.</p>
<h2 id="shimla">Shimla</h2>
<p>I started from Dehradun on the night of 28th of November. Left for Delhi in the middle of the night, after much cajoling and giving reassurances to my mum that I wouldn’t be doing bat-sh*t crazy stuff on my trip. Which was not entirely fulfilled as you will see after some time.</p>
<p>The bus ride was full of surprises as air seeped in through like water from the narrow gapes in the metal sheets. And how could I miss the creaking noise of the suspension. All this amounted to me not getting little or no sleep at all throughout the night. Add it to the amount of cold it was outside.</p>
<p>I was dropped well before my destination citing reasons of excessive fog and traffic. Panic? Rushing to the nearest metro station was the answer. Reached Kashemere Gate from where we had our next bus to Shimla. Devaraj tagged along from there.</p>
<p>At least, this bus was a decent one. I completed on the sleep which I had missed while Devaraj blasted on EDM on his earphones!</p>
<p>We reached quite late. Late by the standards of Shimla! 8:30p.m was the time and we could not see a dog roaming around! The next shocker came when we got to know that the booked hotel was another 2 hour journey from where we were.</p>
<p>No wonder it was cheap when we booked! Heck it was in the middle of the jungle!</p>
<p>We cancelled the reservation and asked the taxi driver(as it was the only means of transport available at that time) to take us to a decent hotel within our specified budget.</p>
<p>250 bucks was what he took for just 2kms. Even airlines charge less than that! Heck, a normal bus ticket from dehradun to delhi costs around 230 INR. And we are looking at a distance of around 240 kms. If it was a time before the sun had set, I would have called it daylight robbery!</p>
<p>The sweet talking manager showed us our rooms which had a great view of shimla from the window. We settled down and changed. He continued babbling about the package which he was providing us. “The cheapest rates in the country”, that’s what he said and I quote it. We said we would think about it and let him know in the morning! So logically, this means we did not accept his terms right? So far so good. Read along.</p>
<center><img src="/content/images/2016/12/room_shimla.jpg" /></center>
<p>We had dinner at the local Chinese cuisine shop after which we roamed around the famous mall road. Handicrafts were priced off the roof. No wonder, given the amount of tourists which flocked this city.</p>
<center><img src="/content/images/2016/12/room_shimla_1.jpg" /></center>
<p>Morning gave us a rude shock. This guy(the same guy who came with to us with his sweet mouth) was trying to dupe us! As we were checking out, he said he had already booked a car for our two day tour of Shimla. The car charges per day being 2.5k INR excluding the room charges. Angry? Oh a lot!</p>
<p>He kept asking 3x the room amount for the car that he had supposedly booked. Now mind you that we are not two drunk dudes who would just randomly say yes to any BS someone would present to us. This day being no different.</p>
<p>After much shouting and drama, we settled with him for the initial booking amount of the room for the day.</p>
<center><img src="/content/images/2016/12/hanuman.jpg" /></center>
<p>Just when we were thinking that this trip was going on a downhill drive, we had our impromptu rafting trip.</p>
<p>And boy it was amazing! 25kms of raw nature. I cannot articulate the feeling. And 3 hours later, after countless rapids and being drenched to the bone, we reached back to the last spot from where we would be picked up back on the trucks. We changed our wet clothes with some dry ones and hopped inside our ride back. I slept like a baby.</p>
<center><img src="/content/images/2016/12/rafting_1.jpg" /></center>
<p>The day ended on a high with some adventure sports at kufri which I missed up on my reluctance to not wake up from my slumber(read: drained out)</p>
<center><img src="/content/images/2016/12/kufri.jpg" /></center>
<p>Luckily the second hotel room which we booked had a nice steam bath. Yes, a steamed bathroom. Add it to Wooden planks and comfy blankets.</p>
<h2 id="manali">Manali</h2>
<p>The 2 nights spent in Shimla went just fine. And on the 3rd day, we left for manali early in the morning.</p>
<center><img src="/content/images/2016/12/manali1.jpg" /></center>
<p>Before I write any further. I confess, that we were one of those guys who didn’t know that kullu and manali were two different places altogether. Hey, I never said that I am good at geography!</p>
<p>So before Manali, you arrive at Kullu if you go by road from Shimla.</p>
<blockquote>
<p>For a change, not everyone was trying to loot us. And it was a little warmer than Shimla. We got a decent room for ourselves and settled for the night.</p>
</blockquote>
<p>Manali was a little over 9kms from our place and we left early the next morning.</p>
<center><img src="/content/images/2016/12/manali2.jpg" /></center>
<p>It was a mix of Shimla’s mall road and valleys that you find around in Pune. We visited the local Tibetan monasteries, bought a lot of souvenirs and some indigenous handicrafts.</p>
<center><img src="/content/images/2016/12/manali3.jpg" /></center>
<p>We returned back to Kullu the same night. ATMs were scarce. Further, finding ATMs which had any money in them was like finding water in a desert. But we survived the night.</p>
<center><img src="/content/images/2016/12/manali5.jpg" /></center>
<p>Kheerganga is a good destination for trekking if you are in the mood for any of it. But as we didn’t plan it, we left it out for another reason(read on)</p>
<center><img src="/content/images/2016/12/manali4.jpg" /></center>
<p>Van vihar should have been prepended with a “big” before it. Minus the fact that it was not snowing, I liked it more than Shimla.</p>
<center><img src="/content/images/2016/12/manali6.jpg" /></center>
<h2 id="kasol">Kasol</h2>
<p>In the similar lines of Shimla, 2 nights was all at kullu.</p>
<center><img src="/content/images/2016/12/kasol1.jpg" /></center>
<p>The next morning, we paid off the hotel manager. And asked him how to reach Kasol. And boy was he happy on hearing that word.</p>
<p>Imagine a tall unshaven guy, wearing a tattered blue denim with a pair of Quechua’s. Pair that along with a black hoodie and a monkey cap not of the ordinary types.</p>
<p>We first had to reach bhunter and then take a bus to kasol. We reached kasol just before sunset and got us a good room. It was a real steal at the price at which we got.</p>
<p>You could find the essense of Israeli culture almost everywhere. The shops, the cafes. All had menu cards, graffiti and wall posters in Israeli.</p>
<center><img src="/content/images/2016/12/kasol2.jpg" /></center>
<p>You ask why was our room deal a good one? Our room was situated right beside the Parvati river. We had the fabled Dhauladhar mountain range right in front of us. Add the fact that we were right in the middle of parvati valley. Picturesque? You bet!</p>
<center><img src="/content/images/2016/12/kasol4.jpg" /></center>
<p>The main Kasol is a small place and you can literally cover it within an hour by foot. All the shops and cafes are situated around a T-shaped area.</p>
<p>This place is a must for anybody who is attracted towards the <code class="language-plaintext highlighter-rouge">hippie culture</code>. You will find Israeli’s buying groceries talking in fluent Hindi. People roam around wearing long beaded jewelry, colorful khaki pants (imagine a purple tinge for example), flower print tops, unkempt dirty long hair(no offence) filled with beads.</p>
<p>Some Beetles and Bob Dylan music and there you are back to the late 1960s hippie culture of the United States.</p>
<center><img src="/content/images/2016/12/kasol3.jpg" /></center>
<h2 id="manikaran">Manikaran</h2>
<p>Manikaran is a little over 7-8kms from Kasol. It was a little bigger than our previous stop. The only place worth visiting here is the Ram temple. It has a natural hot spring in it’s vicinity where you can find people overjoyed of playing inside it!</p>
<center><img src="/content/images/2016/12/manikaran1.jpg" /></center>
<p>And no, we didn’t take a dip inside it. Why? Short on time. I must warn you that the irregular bus timings here will really make you mad.</p>
<p>A one night stay at Kasol was enough for the whole thing!</p>
<center><img src="/content/images/2016/12/manikaran3.jpg" /></center>
<h2 id="dharamshala">Dharamshala</h2>
<p>This trip from Manikaran was the most grueling of bus trips till now.</p>
<p>Manikaran to Kasol was the first leg. This went swiftly. Next stop was Bhunter. After that we had to catch a bus for Mandi. All this consumed a lot more than our patience and time. We reached Mandi just a little after 4o’clock.</p>
<p>Such was our luck that we just missed a direct bus for Dharamshala a few minutes before arriving. We were famished! Our stomachs were empty since the raw Israeli breakfast that we had back at Kasol(it was Devaraj’s idea).</p>
<center><img src="/content/images/2016/12/kasol5.jpg" /></center>
<p>We had some momo’s at a local shop near the bus stand and came back asking for directions.</p>
<p>The alternatives at hand were to either catch a bus for a place near Dharamshala’s vicinity and then pray that we got another bus from there at around 9p.m!</p>
<p>Or we could wait for another direct bus to Dharamshala. The catch here was that we would be reaching there not anytime before 12a.m. Oh, did I mention we stopped believing in any advance room bookings since the Shimla incident?</p>
<p>And we could have always booked a room at Mandi and waited for the next morning and hope that everything would pan out perfectly while we slept our bums out.</p>
<p>Given how reckless we are, we chose the 6o’clock bus. If you have ever gotten into a BEST bus(referring to the local state transport for Bombay), our bus was worse than that. I could hardly position my legs in a comfortable way. Sleeping was out of question. But we had to endure. I mean did we have any other choice here?</p>
<p>Fast forward some uncomfortable hours, I was jolted by a thud at my feet. Next thing I know, a man was literally down at my feet. A tinge of alcohol arose from under his breath. It became apparent that he did not fall down from his seat by the violent jerks which the driver didn’t care giving the passengers.</p>
<p>His fellow mate (or what I assumed him to be) swooped him up from my feet before I kept my bag aside to help him. He placed him back to his seat.</p>
<p>We reached Dharamshala’s ISBT a few minutes shyer than 12:30a.m. In search of a hotel, we started roaming the streets. There was nobody to be found on the streets, Leave alone dogs. Well I didn’t expect anybody to be out of their houses at that unearthly hour given the experience at Shimla where the whole of Mall road closes by 9:30p.m.</p>
<p>After some hurried searching, we got a decent room and settled down. The squeamish feeling since Mandi didn’t go away hence I didn’t want anything to eat. But Devaraj was feeling uncomfortably hungry. He went out in search of some food at a time when most people are having their first cycle of REM sleep!</p>
<p>To both of our surprise, he did find a local <code class="language-plaintext highlighter-rouge">dhaba</code> serving <code class="language-plaintext highlighter-rouge">roti sabzi and dal</code>. While Devaraj was gobbling down his supper, I read on that our honorable Chief Minister, Jayalalitha had passed away that day! News of which came as a shocker for both of us.</p>
<p>Sleep came naturally and both of us greeted our families on phone with a good afternoon as it was past 12 when we woke up the next day.</p>
<h2 id="sidhpur">Sidhpur</h2>
<p>Next stop was the famed Norbulingka institute.</p>
<center><img src="/content/images/2016/12/dharamshala1.jpg" /></center>
<p>I would say it would be the one of the best places for someone who wants to have a closer look at the Tibetan culture, their history and how they live their daily lives.</p>
<center><img src="/content/images/2016/12/dharamshala3.jpg" /></center>
<p>You have workshops which teach you about the traditional wood carving practices.</p>
<center><img src="/content/images/2016/12/dharamshala5.jpg" /></center>
<p>The 3-D wall paintings which you can see on monasteries, the bronze buddha idols which are hand crafted to perfection in the local workshop. It was beautiful!</p>
<center><img src="/content/images/2016/12/dharamshala2.jpg" /></center>
<p>The whole compound resonated rich Tibetan architecture. They had a Museum which had exhibits displaying how different sects of the Buddhist communities dressed. All the way from the common man to the wives of kings and their family.</p>
<center><img src="/content/images/2016/12/dharamshala4.jpg" /></center>
<p>We thought about dropping by the souvenir shop inside to find something in our budget but it was the exact opposite! But I won’t complain as this institute is self sustained and lives off these earnings. Very well maintained. I give them that.</p>
<center><img src="/content/images/2016/12/dharamshala7.jpg" /></center>
<p>We came back to our room and had our dinner after some time and went to sleep.</p>
<center><img src="/content/images/2016/12/dharamshala6.jpg" /></center>
<h2 id="mcleodganj">McLeodGanj</h2>
<p>Way back in the 1930s, McLeodGanj was literally left for ruins when it was hit with a massive earthquake. It was brought back to the scene when the exiled Tibetan govt was given refuge in its hills.</p>
<p>The honorable Dalai Lama and his house of ministers all reside in and around that area. I found that the Tibetan souvenirs, if to be bought, were much cheaper in comparison to kasol/shimla/manali here. This was to be expected as a large population of exiled Tibetan refugees stay here. So if you want to shop, this is the place to do so!</p>
<p>We hopped around some shops and had our refreshments keeping in mind the affordable shops to buy from while returning back. And we left for triund.</p>
<h2 id="triund">Triund</h2>
<p>As planned, our trek to Triund started when we reached McLeodGanj. Heck we had to give up on Kheerganga for this one.</p>
<p>We started at around 12:30p.m and expected the 7 km uphill trek to be completed within 3-4 hours. At least that’s wha t we thought about.</p>
<p>The path to the top of the hill is not a straight climb but full of ridges and small rocks. Mind you we had our rucksacks and in my case it was no less than 10 kilos(thanks a lot to the carefree shopping we did in Manali and elsewhere!) which I had to balance with the body weight.</p>
<p>Loose placed rocks, rubble, dry grass mixed with the dump of local herders cattle. You name it! One slip and down you go.</p>
<center><img src="/content/images/2016/12/triund2.jpg" /></center>
<p>The initial wider and plain track gave way to a much narrow and harder path. We had our moments where we had an extremely beautiful view of McLeodGanj in front of us and there were times when Devaraj would almost collapse (blackout if you may) due to fatigue. He has a heart condition which he didn’t tell me before. But never the less.</p>
<center><img src="/content/images/2016/12/triund3.jpg" /></center>
<p>There were 3-4 shops placed (2 were closed the day we visited) at the edge of the path where you could buy chocolates, water bottles and Maggi. They sold things at 2x the MRP but I think this is justified.</p>
<p>The only means of taking goods up the hill are donkeys. Each donkey costs 800 INR a per trip from the campsite to the hill top.</p>
<center><img src="/content/images/2016/12/triund1.jpg" /></center>
<p>The shops have dustbins where they store all the garbage to be given back to the same donkeys when they come back on a U-turn to be taken downhill. The charges for this is cheap. 9/- INR per bag. Enough with the tariffs.</p>
<p>So we reached the famous <code class="language-plaintext highlighter-rouge">magic view cafe</code>. Well it’s not exactly the typical CCD or barista that we frequent out. But apparently, it’s the oldest <code class="language-plaintext highlighter-rouge">chai</code> point on the way to triund’s top.</p>
<p>After some refreshments and countless pit stops, we did reach the top. I mean finally!</p>
<p>We reached just before the sunset and the view! It was ethereal. The sun rays falling down on the dhauladhar mountain range. The dogs running around and playing with each other while occasionally coming to the lap of the tea stall owner, wagging his tail for some biscuits.</p>
<center><img src="/content/images/2016/12/triund4.jpg" /></center>
<center><img src="/content/images/2016/12/triund6.jpg" /></center>
<p>Pain, fatigue and thirst were long lost friends when we sat down and drank the last few gulps of water left in our bottles. I felt saintly!</p>
<center><img src="/content/images/2016/12/triund11.jpg" /></center>
<blockquote>
<p>Devaraj was as usual puffing on to one of his Marlboro’s while I checked my cell phone for any coverage. Zero signal! But what else do you expect at 2875 meters above sea level!</p>
</blockquote>
<center><img src="/content/images/2016/12/triund8.jpg" /></center>
<p>We got ours tents fixed and settled down our bags in them and got out to join the others in the bonfire. We had a very diverse group of people in our vicinity. Some came from Mumbai, a group of students from Delhi University and some from Banaras.</p>
<center><img src="/content/images/2016/12/triund7.jpg" /></center>
<p>As we devoured down our modest plate of <code class="language-plaintext highlighter-rouge">Dal Chawal</code> (which costed a bomb!) we talked about our experiences and past. Our travel shenanigans. Discussed our plans in life like we were childhood friends reunited after a long time!</p>
<p>Time for sleep came a lot early for me as I was feeling really cold and fatigued. I slept with 3 layers of clothes, a muffler, a bunny cap and a sleeping bag on top of it! And still I was shaking vigorously due to the cold for some time.</p>
<p>So here we were watching “Harry Potter and the deathly hallows” on Devaraj’s phone, sharing his earpieces and finishing the last piece of Lays packet we had on us and before I knew it I was fast asleep.</p>
<blockquote>
<p>I don’t know what the time was, but I woke up to some heavy breathing nearby. I thought it was Devaraj vaping but he was fast asleep.</p>
</blockquote>
<p>Turned out the dog decided to sleep right next to my side of the tent. I could literally feel his hot and heavy breathing, continuously near me. I told Devaraj that a black bear (the chaiwala told us that they frequented the path uphill in search of food sometimes) came to visit us in the morning! And he was gullible enough to even say OMG for a second.</p>
<p>It may be around 6o’clock in the morning, opened the tent sheets and this was in front of us.</p>
<center><img src="/content/images/2016/12/triund9.jpg" /></center>
<p>You will actually have to be there to just feel what it felt like. We had on one side the Dhauladhar mountain range in front us which looked like a golden cone crumpled all the way from the top and on the other side we saw the city of McLeodGanj through the misty clouds.</p>
<p>You might wonder if you have any houses around in the hill top? Well no, just some tea stalls put together with large stones and the tents.</p>
<p>So would there be any bathrooms? Well of course not!</p>
<p>Surprisingly not, there is not even a makeshift loo for you. So what’s the solution?</p>
<blockquote>
<p>You have to figure out a spot for yourself and take a dump. While doing so, you have to hope that nobody comes around in search for their perfect spot. I don’t know how does it compare to taking a dump near a railway track as I haven’t done it.</p>
</blockquote>
<p>The climax comes over when the spot where you are squatting over is on a cliff. So if you accidentally slip over. You are not coming back alive the 2780 meters! Hope you have done your insurance?</p>
<p>After a bowl of Maggi, packing our rucksacks and bading goodbye with the friends we made, we left the summit.</p>
<p>From Dharamshala, Devaraj got on a bus to Delhi and I boarded a bus to Chandigarh which I reached at around 11:30p.m. Mum reminded me again of staying over for the night but I wanted to reach back home badly.</p>
<p>With all the bad decisions so far, this was no better. I reached Dehradun at around 3:30a.m.</p>
<p>No Taxi, no auto. No nothing! Oh no, I didn’t walk all the way to my house!</p>
<p>I did wait for the night to get over at the bus stand. Boarded the first city bus and got off near my house.</p>
<p>Best trip till now for me!</p>
<center><img src="/content/images/2016/12/triund10.jpg" /></center>
<p>So that was much about it. If you are reading till the end. I sincerely thank you for being so patient!</p>
<p>Between, I have hardly pinned any photos for the trip here. So if you wanted a deeper look, <a href="https://instagram.com/tasdikrahman/">my friends on Instagram</a> will surely complain about my never ending feed from this trips photographs!</p>
<p>Godspeed!</p>
Demystifying how imports work in python, ChennaiPy
2016-10-24T00:00:00+00:00
https://www.tasdikrahman.com/2016/10/24/demystifying-how-imports-work-in-python-ChennaiPy
<h2 id="22nd-october-2016">22nd October, 2016</h2>
<p>Clock goes overboard and tries waking me up. Which it has been unsuccessfully trying to do for the last two months. Thanks ma,
for the lovely gift. But I woke up to the sweet melody of my roomate’s snoring.</p>
<p>Anyway, I was still intoxicated by last nights coffee. Remembered I still had to finish the slides of the talk
that I had to give over @ <a href="http://www.meetup.com/Chennaipy/events/234639862/">ChennaiPy, October Meetup’16</a>.</p>
<p>So finally, I did complete the slides and off we went over to the IMSC, chennai campus where the meetup was to be held.
This being my second time there.</p>
<h2 id="deep-dive-into-dictionary">Deep dive into dictionary</h2>
<p>The first talk up was given by the very talented <a href="https://twitter.com/makernaren/">Naren</a>, who works as a Backend engineer @
<a href="http://vue.ai/">Mad street Den</a>. He went on explaining how the interals of dictionary is implemented as Hash tables in
python.</p>
<h2 id="demystifying-how-imports-work-in-python">Demystifying how imports work in python</h2>
<p>Fun fact. This was the talk for which I had been preparing the slides for earlier that morning. All in all, it was a very good
feeling to give a talk in front of a large crowd.</p>
<p>Here are the slides.</p>
<script async="" class="speakerdeck-embed" data-id="df1b0dd2c89b44678015f3565c876881" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<p><strong>Code snippets:</strong> <a href="https://github.com/tasdikrahman/talks">https://github.com/tasdikrahman/talks</a></p>
<h2 id="introduction-to-selenium">Introduction to selenium</h2>
<p>Mayur did a great job in introducing us to testing using selenium. Was a fun talk overall.</p>
<center><img src="/content/images/2016/10/chennaipy1.jpg" /></center>
<p>That’s me by the way. Thanks to the bad camera quality <em>sigh</em>.</p>
<center><img src="/content/images/2016/10/chennaipy2.jpeg" /></center>
<center><img src="/content/images/2016/10/chennaipy3.jpeg" /></center>
<center><img src="/content/images/2016/10/chennaipy5.jpeg" /></center>
<p>The lovely crowd for the meetup. Cheers to them!</p>
Pycon India 2016, New Delhi
2016-10-19T00:00:00+00:00
https://www.tasdikrahman.com/2016/10/19/Pycon-India-2016
<p>This year being no different, I attended PyCon India (yet again, this being my 3rd one). The only difference being that this time it was
being held at New Delhi instead of Bangalore.</p>
<p>Met many of my old friends, made some new ones, interacted with some really interesting people and should I say met some legendary guys/gals too.
All in all it was just like the previous year. Felt just like home!</p>
<h2 id="talks">Talks</h2>
<p><a href="https://twitter.com/punchagan">Puneet</a> gave a really interesting talk about how to generate tests and not actually write them. <a href="https://twitter.com/anandology">Anand</a> as usual was on his best and his talk on
decorators was truly delightful. All in all, every talk had something to take away home with.</p>
<p>On top of it, we had <a href="https://amueller.me/">Andreas Mueller</a> with us this time. He is a core contributer and maintainer to the <a href="http://scikit-learn.org/">scikit-learn</a>
machine learning library.</p>
<p><a href="https://in.linkedin.com/in/levkonst">Lev</a> was there too. He is the community manager for the <a href="https://radimrehurek.com/gensim/">gensim library</a>. <a href="https://twitter.com/ghoseb">BG</a> from helpshift. <a href="https://www.linkedin.com/in/vanlindberg">Van</a> from Rackspace International.
And the list just goes on!</p>
<h2 id="ending-note">Ending note</h2>
<p>I could write another two paragraphs about it but as I have my mid sems starting tomorrow, I will keep it short and try opening those notes
I borrowed.</p>
<p>Some pictures to drool by.</p>
<center><img src="/content/images/2016/10/pycon1.jpg" /></center>
<center><img src="/content/images/2016/10/pycon2.jpg" /></center>
<p>If you are looking for me, I am the unkempt haired guy with the black wingify T-shirt</p>
<center><img src="/content/images/2016/10/pycon3.jpg" /></center>
<p>Pycharm spreading some of their love</p>
<center><img src="/content/images/2016/10/pycon4.jpg" /></center>
<p>That was Lev and his Random Forest T. ML guys would get the reference here.</p>
<center><img src="/content/images/2016/10/pycon5.jpg" /></center>
<p>And that is Van. What a humble guy. Given that he is a Vice President @ RackSpace.</p>
<p>The only thing which I regretted was to not propose a talk for this years PyCon. But I hope I will do so for the next years PyCon.
See you there then!</p>
My internship experience at Wingify (VWO team), New Delhi
2016-08-13T00:00:00+00:00
https://www.tasdikrahman.com/2016/08/13/My-internship-experience-at-Wingify-new-delhi-visual-website-optimiser-vwo
<p>As I am sitting here at the Delhi Airport waiting for my flight back to Chennai, I could just not stop myself from thinking about my time as an intern back at Wingify which ended last week. Here’s what I wrote down after getting carried way with several cups of coffee (thanks for luring me with that smell costa coffee)</p>
<p>So here it is then!</p>
<h2 id="day-1">Day 1</h2>
<p>It was 5 o’clock in the morning and I was quite drowsy. Reason being the all nighter I pulled the other night for the last exam of our end sems. <strong>phew</strong></p>
<p>Here I was at the Delhi airport just a day after my semester exams, ready to start with my internship. Talk about eagerness here!</p>
<p>A small part of me was also happy that I was moving out of Chennai! (at least for some time)</p>
<p>Joined them the next day in their main office at the heart of NSP, Delhi.</p>
<center><img src="/content/images/2016/08/wingify_rig_mine.jpg" /></center>
<p>Now I was naturally excited to work in a company which had grown and become one of the best startups in India in such a short span of time. On top of that, this was my first internship in a well-established product based start-up and I was hoping that I could learn all that I could and perform in accordance to their standards.</p>
<center><img src="/content/images/2016/08/wingify_officespace_2.jpg" /></center>
<p>I was introduced to Ankit Jain (Lead software Engineer at Wingify) and Ajay Sharma (Senior Software Engineer) by my HR. We had a brief chat where I was told I would be working with the Backend Development team for VWO, their flagship product.</p>
<center><img src="/content/images/2016/08/wingify_officespace_1.jpg" /></center>
<p>Talking about VWO, it’s the world’s easiest A/B testing tool. And we are quite good (read “The Best”) at it! The month before I joined, we had monthly sales crossing a little over 1 million dollars.</p>
<p>After getting up and ready with my development environment, I was given my first project.</p>
<h3 id="integration-of-statsd-and-graphite-project-1">Integration of Statsd and graphite (Project #1)</h3>
<p>StatsD collects and aggregates metrics and then ships them off to Graphite which stores the time-series data and enables us to render graphs based on these data.</p>
<p>Graphite consists of three parts.</p>
<ul>
<li>
<p>carbon: a daemon that listens for time-series data.</p>
</li>
<li>
<p>whisper: a simple database library for storing time-series data.</p>
</li>
<li>
<p>webapp: a (Django) webapp that renders graphs on demand.</p>
</li>
</ul>
<p>The setting up of the the overall stack was a bit archaic but I finally got it right and the metrics for our internal service were being graphed correctly by Graphite. And they looked pretty too!</p>
<p>Coming back, the service on which we integrated StatsD and graphite runs on several servers. So while plotting the graphs we wanted to know the server from where the stats are being pushed on to the buckets of statsd. Well that was much about it.</p>
<h3 id="bumblebee---an-experimental-slack-bot-vwo-project-2">Bumblebee - An experimental slack bot VWO (Project #2)</h3>
<p>Wingify has this culture of organizing hackathons at the end of every month, where people from the engineering team come together to hack on something which they want to see at VWO.</p>
<p>To be honest, I was quite clueless on what to build for the first half an hour or so and after a little nudge from Ankit I decided upon bumblebee. Bumblebee makes use of the beautiful VWO API to provide functionalities (if not all) to the VWO account holder right at the comfort of his slack channel. Like you can get details of all the campaigns of your account, check their status (whether they are running, paused et el). Update status to Start/Stop/Pause a particular campaign. Share your campaign with someone else and some more things.</p>
<p>It was written in python and Ankit was too kind to let me open source it. Here is the link for the curious.</p>
<p><a href="https://github.com/wingify/bumblebee">https://github.com/wingify/bumblebee</a></p>
<h3 id="optimization-much-project-3">Optimization much? (Project #3)</h3>
<p>My 3rd project revolved around optimization of an internal service. I had to increase the efficiency (read performance). I implemented some rough 3 approaches and the last one bumped the performance by up to 23.6%. I could have tried for dropping it down further to a lower one but sadly the end to my internship was looming around the corner so I dropped it. And that was the 3rd and the last project I did as an intern at Wingify.</p>
<h2 id="how-was-my-experience">How was my experience?</h2>
<p>My experience? I loved it there!</p>
<ul>
<li>
<p>Solving hard engineering problems. Check</p>
</li>
<li>
<p>Extremely talented engineering team. Check</p>
</li>
<li>
<p>Approachable mentors. Check</p>
</li>
</ul>
<p>Heck, here I am with Abhishek Batra, the Android guru here and Abhishek Garg, our DevOps pro. Turns out that we all came along quite well along each other along with the other people and yes these guys are quite senior to me :)</p>
<center><img src="/content/images/2016/08/wingify_tasdik_abhishek.jpg" /></center>
<ul>
<li>
<p>Awesome Work Culture. Check</p>
</li>
<li>
<p>Delhi :P. Check</p>
</li>
</ul>
<p>Jokes apart. I made some really good friends back there and learned a ton from everyone. I am proud that I was part of a team which is building something which people love and has an impact on thousands of customers.</p>
<h2 id="so-what-now">So what now?</h2>
<p>Looking back at the time when I received a call from Nupur about my acceptance as an intern at Wingify. I was thinking about whether to join it over the other 6-7 odd companies which accepted my application as a summer intern.</p>
<center><img src="/content/images/2016/08/wingify_table_tennis.jpg" /></center>
<p>After the two months that I have spent here at Wingify, I now believe that I did just the right thing on choosing Wingify over others!</p>
<p>And did I mention that they gave me a pre-placement offer :) ?</p>
<p>This was taken on my last day at office. And boy was I sad!</p>
<center><img src="/content/images/2016/08/wingify_newdelhi_tasdik.jpg" /></center>
<p>Ankit threw a huge party at a pub in Rajouri garden for all the interns along with the other guys and my last day turned out to be the best day in Delhi.</p>
<center><img src="/content/images/2016/08/wingify_2016_summer_interns.jpg" /></center>
<p>Until next time Delhi!</p>
<p><em>This post was cross posted at <a href="http://team.wingify.com/tasdik-talks-about-his-internship-experience-at-wingify">Wingify’s team blog too</a></em></p>
Decorators 101 - A gentle introduction
2016-07-21T00:00:00+00:00
https://www.tasdikrahman.com/2016/07/21/Decorators-101
<h2 id="decorators-you-say">Decorators you say</h2>
<p>If you are familiar with <code class="language-plaintext highlighter-rouge">python</code>, chances are that you might have already seen the decorator syntax. It comes off as a simle
concept when being used, but when you try to get your head around the underlying details, you find yourself in a hot fix. And are probably
asking yourself</p>
<blockquote>
<p>How the heck does it work?</p>
</blockquote>
<p>Python does a very good job in abstracting about the intricacies, so much so that we take it almost for granted. Remember the
routes in <code class="language-plaintext highlighter-rouge">flask</code>?</p>
<p>Adding a route is as simple as doing a</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">route</span><span class="p">(</span><span class="s">'/index/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello</span><span class="p">():</span>
<span class="k">return</span> <span class="s">"hello there"</span>
</code></pre></div></div>
<p>Where the <code class="language-plaintext highlighter-rouge">@</code> symbol denotes the decorator syntax.</p>
<p>But before diving into decorators, discussing about functions would seem appropriate</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"hello {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">foo</span>
<span class="o"><</span><span class="n">function</span> <span class="n">foo</span> <span class="n">at</span> <span class="mh">0x7f59dc601aa0</span><span class="o">></span>
<span class="o">>>></span> <span class="n">foo</span><span class="p">(</span><span class="s">"tasdik"</span><span class="p">)</span>
<span class="s">'hello tasdik'</span>
<span class="o">>>></span> <span class="n">bar</span> <span class="o">=</span> <span class="n">foo</span>
<span class="o">>>></span> <span class="n">bar</span>
<span class="o"><</span><span class="n">function</span> <span class="n">foo</span> <span class="n">at</span> <span class="mh">0x7f59dc601aa0</span><span class="o">></span>
<span class="o">>>></span> <span class="n">bar</span><span class="p">(</span><span class="s">"body double"</span><span class="p">)</span>
<span class="s">'hello body double'</span>
<span class="o">>>></span>
</code></pre></div></div>
<p>If you have been dabbling away in python, this might not seem very unfamiliar to you I assume.</p>
<blockquote>
<p>As you can see, functions can be assigned to each other</p>
</blockquote>
<p>Everything in python is considered as a first class object, which include <code class="language-plaintext highlighter-rouge">functions</code>, <code class="language-plaintext highlighter-rouge">classes</code> and everything else which you thought
could not be an object. Jokes apart, this paradigm is really different from the other programming languages but it has it’s own
advantages to it.</p>
<p>Moving forward this analogy of treating everything in python as first class objects. Functions can also be passed on other functions!
Not sure about that? Here you go</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">sum</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">a</span><span class="o">+</span><span class="n">b</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">diff</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">a</span><span class="o">-</span><span class="n">b</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">operation</span><span class="p">(</span><span class="n">func</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">operation</span><span class="p">(</span><span class="nb">sum</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="mi">30</span>
<span class="o">>>></span> <span class="n">operation</span><span class="p">(</span><span class="n">diff</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="o">-</span><span class="mi">10</span>
<span class="o">>>></span>
</code></pre></div></div>
<p>We have passed around the function name to the function <code class="language-plaintext highlighter-rouge">operation()</code> as we pass around normal values.</p>
<p>Now what if I told you functions can be returned as return values for other functions! Let’s see how we do that</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">value</span><span class="p">):</span>
<span class="p">...</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">2</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">bar</span><span class="p">():</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">value</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">bar</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">bar</span>
<span class="o"><</span><span class="n">function</span> <span class="n">foo</span><span class="p">.</span><span class="o"><</span><span class="nb">locals</span><span class="o">></span><span class="p">.</span><span class="n">bar</span> <span class="n">at</span> <span class="mh">0x7fe2c7229b70</span><span class="o">></span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">bar</span> <span class="o">=</span> <span class="n">foo</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">bar</span><span class="p">.</span><span class="n">__closure__</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">cell_contents</span>
<span class="mi">10</span>
<span class="o">>>></span> <span class="n">bar</span><span class="p">()</span>
<span class="mi">10</span>
<span class="o">>>></span>
</code></pre></div></div>
<p>Not that’s surprising then I suppose. The only gotcha here would be that the inner enclosing functions have the access to the
enclosing function variables. That is the reason, the variable <code class="language-plaintext highlighter-rouge">value</code> is still accessible to the function <code class="language-plaintext highlighter-rouge">foo()</code></p>
<p>The <code class="language-plaintext highlighter-rouge">foo</code> function displays the closure property beautifully here as it stores the value that was passed on to it.</p>
<p>Talking about closures, this can be used very cleverly in some cases</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">val1</span> <span class="o">=</span> <span class="n">foo</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">val1</span>
<span class="o"><</span><span class="n">function</span> <span class="n">bar</span> <span class="n">at</span> <span class="mh">0x7f59dc601de8</span><span class="o">></span>
<span class="o">>>></span> <span class="n">val1</span><span class="p">()</span>
<span class="mi">10</span>
<span class="o">>>></span> <span class="n">val2</span> <span class="o">=</span> <span class="n">foo</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">val2</span><span class="p">()</span>
<span class="mi">20</span>
<span class="o">>>></span>
</code></pre></div></div>
<p>You can see a special behaviour here demonstrated by the function <code class="language-plaintext highlighter-rouge">foo()</code>. It remembers the value passed to it between function
calls. A property which can then be utilized for implementing other features. It’s somewhat similar to the demonstration of public and
private interface. Where the function <code class="language-plaintext highlighter-rouge">foo()</code> would be acting as the public function and <code class="language-plaintext highlighter-rouge">inner()</code> being the private one.</p>
<h2 id="so-how-do-i-write-one">So how do I write one</h2>
<p>Simply put, decorators are nothing but funcions which take on another functions and modify it’s behaviour without changing the original
code</p>
<p>Confused? Let’s write one</p>
<p>This is more useful in the context when we have a function and we want to modify the output of that function without playing around
with the original source code. Reasons may be that we are not allowed to do or because it’s simply not possible, whatever the reason
may be. Decorators are here to the rescue.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"hello there {}!"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">tagify</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"<p>{}</p>"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">wrap</span>
<span class="p">...</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">tagify_tasdik</span> <span class="o">=</span> <span class="n">tagify</span><span class="p">(</span><span class="n">greet</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">tagify_tasdik</span><span class="p">(</span><span class="s">"tasdik"</span><span class="p">)</span>
<span class="s">'<p>tasdik</p>'</span>
</code></pre></div></div>
<p>So we just decorated the return value of a function!</p>
<h2 id="but-where-is-that--syntax-you-were-talking-all-along">But where is that <code class="language-plaintext highlighter-rouge">@</code> syntax you were talking all along?</h2>
<p>Don’t worry, here is an example for you. Keeping in mind what we have discussed so far. Keeping the above example in mind,</p>
<p>We don’t always have to do <code class="language-plaintext highlighter-rouge">tagify_tasdik = tagify(greet)</code> for decorating our function. Python provides some <code class="language-plaintext highlighter-rouge">syntactic sugar</code>
for doing the same.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">tagify</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"<p>{}</p>"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">func</span><span class="p">(</span><span class="n">name</span><span class="p">))</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">wrap</span>
<span class="p">...</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="o">@</span><span class="n">tagify</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"hello there {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="p">...</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">greet</span><span class="p">(</span><span class="s">"foo"</span><span class="p">)</span>
<span class="s">'<p>hello there foo</p>'</span>
<span class="o">>>></span>
</code></pre></div></div>
<h2 id="chaining-one-or-more-decorators">Chaining one or more decorators</h2>
<p>As the title suggests, let’s say we want to decorate our function further, we can chain the decorators to get the desired output.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">def</span> <span class="nf">p_tagify</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">content</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"<p>{}</p>"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">func</span><span class="p">(</span><span class="n">content</span><span class="p">))</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">wrap</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">h1_tagify</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">content</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"<h1>{}</h1"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">func</span><span class="p">(</span><span class="n">content</span><span class="p">))</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">wrap</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">div_tagify</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">wrap</span><span class="p">(</span><span class="n">content</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"<div>{}</div>"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">func</span><span class="p">(</span><span class="n">content</span><span class="p">))</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">wrap</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="o">@</span><span class="n">div_tagify</span>
<span class="p">...</span> <span class="o">@</span><span class="n">h1_tagify</span>
<span class="p">...</span> <span class="o">@</span><span class="n">p_tagify</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"hello there {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">greet</span><span class="p">(</span><span class="s">"tasdik"</span><span class="p">)</span>
<span class="s">'<div><h1><p>hello there tasdik</p></h1</div>'</span>
<span class="o">>>></span>
</code></pre></div></div>
<p>But wait a second! What do we have here</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">greet</span><span class="p">.</span><span class="n">__name__</span>
<span class="s">'wrap'</span>
<span class="o">>>></span>
</code></pre></div></div>
<p>As you can see, the functions name got changed to the method which was decorating it and this can cause a huge pain when
you are debugging your programs.</p>
<p>But as usual, we have <code class="language-plaintext highlighter-rouge">functools</code> to the rescue</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span>
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">p_tagify</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="p">...</span> <span class="o">@</span><span class="n">wraps</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">decorate</span><span class="p">(</span><span class="n">content</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"<p>{}</p>"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">func</span><span class="p">(</span><span class="n">content</span><span class="p">))</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">decorate</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="o">@</span><span class="n">p_tagify</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"hello there {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">greet</span><span class="p">(</span><span class="s">"tasdik"</span><span class="p">)</span>
<span class="s">'<p>hello there tasdik</p>'</span>
<span class="o">>>></span> <span class="n">greet</span><span class="p">.</span><span class="n">__name__</span>
<span class="s">'greet'</span>
<span class="o">>>></span>
</code></pre></div></div>
<h2 id="passing--arguments-to-decorators">Passing Arguments to decorators</h2>
<p>Now wouldn’t it have been real nice if you could pass on arguments to decorators to tagify the content as you wished. This would
reduce 3 functions into 1. (Remember the decorator chaining example?)</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">wraps</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="k">def</span> <span class="nf">tag</span><span class="p">(</span><span class="n">tag_name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">tag_decorator</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="p">...</span> <span class="o">@</span><span class="n">wraps</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">func_wrapper</span><span class="p">(</span><span class="n">content</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"<{0}>{1}</{0}>"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">tag_name</span><span class="p">,</span> <span class="n">func</span><span class="p">(</span><span class="n">content</span><span class="p">))</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">func_wrapper</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">tag_decorator</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="o">@</span><span class="n">tag</span><span class="p">(</span><span class="s">"p"</span><span class="p">)</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="s">"hello there {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">greet</span><span class="p">(</span><span class="s">"tasdik"</span><span class="p">)</span>
<span class="s">'<p>hello there tasdik</p>'</span>
<span class="o">>>></span> <span class="n">greet</span><span class="p">.</span><span class="n">__name__</span>
<span class="s">'greet'</span>
<span class="o">>>></span>
</code></pre></div></div>
<p>So I hope you now have a good idea about how decorators work in python.</p>
Margo: An opiniated Slack Bot for SRMSE's Slack channel
2016-06-25T00:00:00+00:00
https://www.tasdikrahman.com/2016/06/25/Margo-An-opiniated-Python-based-slack-bot-for-SRM-Search-Engine
<h2 id="bots-before-and-now">Bots: Before and Now</h2>
<p>When was the last time you were having a conversation with a computer before? Most of us will come into the category where we haven’t done so or maybe you did but you didn’t like the experience! The days aren’t far when customer support will be provided by dedicated bots built using cutting edge ML techniques and backed by state of the art NLP research.</p>
<p>If you have been following the latest trends in software industry. Bots and VR are <em>the thing</em> as echoed by a lot of big shot companies and tech evangelists. I mean just look at <a href="https://techcrunch.com/2016/04/07/rise-of-the-bots-x-ai-raises-23m-more-for-amy-a-bot-that-arranges-appointments/">these</a> <a href="https://techcrunch.com/2016/03/17/facebooks-messenger-in-a-bot-store/">crazy</a> <a href="https://techcrunch.com/2016/05/11/kik-already-has-over-6000-bots-reaching-300-million-registered-users/">articles</a> <a href="https://techcrunch.com/2016/05/10/facebook-chatbot-analytics/">on</a> <a href="https://techcrunch.com/2016/05/07/bots-messenger-and-the-future-of-customer-service/">techcrunch</a>. Bots have been the talk of the town since some time now and developers are taking a fair advantage of this rise in attention for promoting their own bots in the market.</p>
<h2 id="what-got-me-into-bots">What got me into bots</h2>
<p>Well to be honest. I am quite a regular to <a href="https://techcrunch.com/">techcrunch’s</a> website. And when I got to know that they had released a <a href="https://telegram.org/">telegram</a> bot for their service, it didn’t take me too long to setup the bot for my telegram ID.</p>
<p>I was quite cynical at first about how the experience would turn out. But surprisingly! The interface was really intuitive and didn’t make any assumptions on their side about what the user knew or didn’t.</p>
<p>The bot helps you stay on top of the topics, stories and people you care about the most. You can subscribe to different topics, authors or sections of the site, and the bot will send you news articles on a daily basis or when you explicitly ask for it.</p>
<p>In short I was really impressed and didn’t regret installing the bot.</p>
<p><img src="https://tctechcrunch2011.files.wordpress.com/2016/03/mar-15-2016-1020.gif" alt="techcrunch" /></p>
<p><em>image courtesy: TechCrunch</em></p>
<h2 id="so-whats-the-deal-with-margo">So what’s the deal with Margo</h2>
<p>Back at <a href="http://srmsearchengine.in/">SRM Search Engine</a>, we manage our own single dedicated server for providing our search service. And sometimes, it goes down due to some or the other reasons. Sometimes for the insane amount of data that we crunch on it or when we are playing around with a new technology which requires some downtime.</p>
<p>The idea for <a href="https://github.com/tasdikrahman/margo">Margo</a> came to me while having a chat with one of my college mates on our slack channel. Oh no, we were not talking about bots! Something totally different, but well it came out of the blue to me!</p>
<p>Weekend ahead! Weather in Delhi (ok. it was raining that day!), along with some aloo tikkas. What a bliss!</p>
<p>Reading the docs for the <a href="https://api.slack.com/">Slack API</a> simultaneously and working on my bot. I had a working prototype ready in about 2 hours. The next 1 hour was spent refactoring the app.</p>
<p>Here’s a glimpse to what it does</p>
<p><img src="https://raw.githubusercontent.com/tasdikrahman/margo/master/assets/demo.gif" alt="Margo demo" /></p>
<p>Pretty basic for now. I plan to automate the pinging process. But the current deployment of the bot forbades me on doing so. You see, I have deployed it to a basic dyno on <a href="https://heroku.com/">heroku</a>. The thing is that, the dyno goes to sleep if it does not recieve any <code class="language-plaintext highlighter-rouge">HTTP requests</code> afer some time. Moreover, they have a fixed 6 hour downtime for any basic dyno. So yeah, as I am pretty much broke right now. I cannot afford a Digital Ocean/Linode server. But I mean what the heck right, at least it works for now!</p>
<p>More functionalities are coming through over.</p>
<blockquote>
<p>Here’s a link to the project : <a href="https://github.com/tasdikrahman/margo">Margo</a></p>
</blockquote>
<p>All in all, I enjoyed building <a href="https://github.com/margo/">Margo</a> for this was a side project after some weeks (several if you may) of break. Reason being, I hadn’t had much time to indulge in side projects due to my commitments as an intern at <a href="http://wingify.com/">Wingify</a>. And man, I am loving it here!</p>
<hr />
<p>Back there, I just finished working on an internal backend service which is a <a href="https://www.rabbitmq.com/">rabbitMQ</a> consumer handling the consumed messages (based on type of the queue) from the numerous queues and processing them accordingly. Did some refactoring of the service and then integrated <a href="https://statsd.readthedocs.io/en/v3.2.1/index.html">statsd</a> with it to graph the IO operations done by it. <a href="https://graphite.readthedocs.io">Graphite</a> along with it’s components <a href="https://graphite.readthedocs.io/en/latest/carbon-daemons.html">carbon</a> and <a href="https://graphite.readthedocs.io/en/latest/whisper.html">whisper</a> were used to visualize the data in a human readable graph format. Last step was to deploy the setup to a test server on Digital Ocean.</p>
<p>Maybe what I just wrote is quite abstract. But I plan on writing a blog post about my experience with, but let’s see when I get time to write about it!</p>
<hr />
<p>So this weekend, me and some of my friends header over to <a href="http://meetup.com/Bot-Builder-Delhi/">Bot builder workshop meetup, Delhi</a> for the fun of it. We had Beerud Sheth, the CEO of Gupshup give a talk about how bots were the next big thing.</p>
<p>All in all we had a really good time and met some really interesting guys.</p>
<p><img src="https://raw.githubusercontent.com/tasdikrahman/www.tasdikrahman.com/gh-pages/content/images/2016/6/gupshup.jpg" alt="Gupshup" /></p>
<p><img src="https://raw.githubusercontent.com/tasdikrahman/www.tasdikrahman.com/gh-pages/content/images/2016/6/innova8.jpg" alt="innov8" /></p>
<p>Cheers!</p>
Simple lessons learned while building things - My open source journey so far
2016-04-24T00:00:00+00:00
https://www.tasdikrahman.com/2016/04/24/Simple-lessons-learned-while-building-things
<h2 id="reinventing-the-wheel-is-sometimes-a-good-idea">Reinventing the wheel is sometimes a good idea</h2>
<p>One of the stock critiques for any new project is that it’s been done before. You’re working on a new module, format, etc: what about this existing format?</p>
<ol>
<li>
<p>Contributing to existing projects is often impossible if your vision is different from those of the maintainers, your changes are too large, or they’re absent.</p>
</li>
<li>
<p>Even if a problem is “solved” by an existing project, it can often be solved better, faster, in fewer lines of code, or with more documentation.</p>
</li>
<li>
<p>Sometimes writing the thing is the best way to learn about the solution. It’s best to develop the skill of reading code and understanding it, but usually the existing project is not written clearly.</p>
</li>
</ol>
<h2 id="few-open-source-projects-attain-escape-velocity">Few open source projects attain escape velocity</h2>
<p>Most projects attract neither users nor contributors and are forgotten within a year of their development.</p>
<p>Some lucky projects acquire users.</p>
<p>Only the very luckiest projects get contributors, <strong>probably less than 1% of projects</strong>. These have more than one maintainer who really feels continuing ownership and has the time to contribute, and the open source dream can be achieved.</p>
<p>Most projects will cost you time. Your responsibilities to maintain them and their users will add up. The more things you create, the more time you spend helping users and doing their bidding, and less time you have to create new things.</p>
<h2 id="the-ultimate-skill-is-knowing-what-to-do">The ultimate skill is knowing what to do</h2>
<p>Your ability to write code quickly or answer issues or anything else doesn’t matter if you don’t know what to do, and people don’t start out knowing what to do. The experienced builder knows what to prioritize when, and this is what make them effective.</p>
<p>Optimize for performance when you need to. Write minimal docs and expand them once the project is ready. Avoid implementing anything that isn’t necessary. If you expand the mission of a project, do so knowing that it’ll cost time and could sacrifice focus.</p>
<p>Most importantly, don’t build things that are impossible or that you’re sure will turn out bad just because that’s what the plan entails: change the plan and build possible, good things.</p>
<h2 id="read-more-than-you-write">Read more than you write</h2>
<p>Learn to dig into codebases, read code, and understand other people’s styles, and you’ll save yourself thousands of annoying back-and-forth conversations.</p>
<p>Research is underrated. Computer Science history is short, but many problems have established, proven solutions. Knowing those solutions and the principles with which they were solved will give you the power of the ancients.</p>
<p>Here’s a link to my github profile for the curious <a href="https://github.com/tasdikrahman">@tasdikrahman</a></p>
<p>Until next time then!</p>
Extraction of text from image using tesseract-ocr engine
2016-04-04T00:00:00+00:00
https://www.tasdikrahman.com/2016/04/04/extraction-of-text-from-image-using-tesseract-ocr-engine
<p>This post was long overdue!</p>
<p>We have been working on building a food recommendation system for some time and this phase involved getting the menu items from the menu images</p>
<p>We poured over at zomato’s site looking for menu’s and all we found was images in the name of menu’s</p>
<center><img src="http://www.tasdikrahman.com/content/images/2016/4/menu_img.jpg" /></center>
<p>This is not what we wanted!</p>
<blockquote>
<p>We want the menu items tp be in text format, so that we can easily track which restaurants are serving which dish and analyze the reviews to see which restaurant serves best</p>
</blockquote>
<h3 id="scrape-them-all">Scrape them all!</h3>
<p>The first step was to scrape the images of the hotels.</p>
<p>The very first apps which came to our mind when we thought about food were no other than <a href="https://zomato.com">zomato</a> and <a href="http://burrp.com">burrp</a> (brownie points for those for whom these were the names echoing in their minds).</p>
<p>Zomato kept blocking our <code class="language-plaintext highlighter-rouge">crawlers</code> from time to time. So we found <strong>burrp</strong> to be a boon in this sense.</p>
<blockquote>
<p>You can find some of the webscrapers here <a href="https://github.com/foodoh/web_scrapers">(https://github.com/foodoh/web_scrapers)</a></p>
</blockquote>
<h3 id="enter-ocr">Enter OCR</h3>
<p>So the obvious choice was to apply <code class="language-plaintext highlighter-rouge">image processing</code> techniques so as to extract the text inside these images</p>
<p>We thought we would be getting okayish results using the <a href="https://github.com/tesseract-ocr">tesseract-ocr engine</a> for this purpose. If you haven’t heard about it. <code class="language-plaintext highlighter-rouge">tesseract</code> is maitained by <a href="http://google.com">google</a> and provides a decent API for getting the job done!</p>
<p>We ran some of our images on it wihout any pre-processing and waited for the result. But we were in for a rude shock. Not only were we getting bad results but some of them were outright garbage text</p>
<p>Tell me if you can comprehend any of this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> M hm mkflfiffi {MWfl m lax-g;
lug-BI I 1. I I.“ n I
III"- - ‘I I Mb. I I
I‘M“ - ‘I I “IQ-h I .
in“ . 1. I “my I I
III-“mun. - 1I I I‘M n I
lOl-I I I Ila-Inn." P! I
Imus-n I Ia—l— I I
'wm ll Intimi— I I
mm lulu - I III-m - I
'60.“ II I II._~ . o.
'm“ : Iain—Lita 1| I III—I.— - .-
uhfinI—II I . I I w“ n I
nun—n.- . "h." III-nun— a I
m um:
I n
H .
C -
I I
I n
C I
I 1!
</code></pre></div></div>
<p>This is supposed to be the text list of menu items extraced from this image</p>
<center><img src="http://www.tasdikrahman.com/content/images/2016/4/bad_img_ocr.jpg" /></center>
<p>Sucks right?</p>
<p>But some results were turning out fine. Take for instance this image <a href="https://github.com/foodoh/ocrd_menus/blob/master/menu_images/1947-fine-indian-cuisine-banashankari-listing/1947-fine-indian-cuisine-banashankari-listing_1.jpg">(link)</a></p>
<center><img src="http://www.tasdikrahman.com/content/images/2016/4/good_img_ocr.jpg" /></center>
<p>Result for this <a href="https://github.com/foodoh/ocrd_menus/blob/master/tesseract_menu_data/first_600_all_menus/1947-fine-indian-cuisine-banashankari-listing.txt">(link)</a></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>777 fl
SOUP
Tomato Soup 80
Sweet Corn so
SHURUWAAT
Paneer Tirang! Tikka 210
Paneer Tikka La] Mirch 2 w
Paneer Malai Tikka 210
Paneer Kali Mirch 210
Pafieer Peshawari 210
Tandoori Baby Corn 190
Chalpali Baby Corn 190
Alan Nazakat Ke 170
Chaman Kl Seekh 180
A100 Ke Tukde 17o:
Makai Ki mm 190
Kuuiiniri Seekh lao
Tandoori Kumbh 2w
Lahnri Suhzi Seekh 180
Veg Kui-kurc 190
Makai Maxi Scckh 190
— SUBZI KI BAHAAR
A100 Gobi Adarakhi 170
Pindi Choic 11o
Bhendi Do Pyazza 170
Sarson Ka Saag 170
Shin-i121 W313 Emma 170
Had Makai Khas 180
Lasnoni Vegeiahle 180
Lagaan Ki Subzi 180
Kashmiri Dum A100 180
Veg Angare 180
Subzi Zaykcdar 180
Vegomble Lahari 12m
Diwan] Handi 130
Vegetable Kali Mirch 180
Suhzi La Jawab 190
Sahzi Jam Pahechani 190
Jafrani Kufta 190
Panzer Knthmari Kufia 190
</code></pre></div></div>
<p>Decent enough for me.</p>
<h3 id="grayscaling-the-images">Grayscaling the images</h3>
<p>Now after some reading, we found out that grayscaling the images according would increase the OCR accuracy.</p>
<p>A simple <code class="language-plaintext highlighter-rouge">PIL</code> program for that</p>
<script src="https://gist.github.com/tasdikrahman/06239ec6986ce3d05b4dfd00cc038372.js"></script>
<p>This improved the accuracy to a certain extent. Here is a sample greyscaled image for you <a href="https://github.com/foodoh/scraped_menu_items/blob/master/light_cleaned_images/100-ft-boutique-bar-restaurant-indiranagar-listing/100-ft-boutique-bar-restaurant-indiranagar-listing_1_cleaned.jpg">(link)</a></p>
<center><img src="http://www.tasdikrahman.com/content/images/2016/4/greyscaled_img.jpg" /></center>
<blockquote>
<p>Some of the cleaning scripts lie here <a href="https://github.com/foodoh/image_cleansing/">(https://github.com/foodoh/image_cleansing/)</a></p>
</blockquote>
<h3 id="automating-the-task-of-ocr">Automating the task of OCR</h3>
<p>Now tesseract was provinding a <code class="language-plaintext highlighter-rouge">CLI</code> interface for interacting with it. But how would you automate this? I am not gonna sit there and type</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">$ </span>tesseract myscan.png out
</code></pre></div></div>
<p>for each and every image scraped!</p>
<blockquote>
<p>Enter python!</p>
</blockquote>
<p>As always. <code class="language-plaintext highlighter-rouge">python</code> comes to the rescue. I wrote a simple script which ran over the image directories, looping over each and every image for each hotel and ran <code class="language-plaintext highlighter-rouge">tesseract-ocr</code> on them.</p>
<p>Storing of each hotel’s text menu was done in a different file with the name that file being the hotel’s normalized name.</p>
<p>You can find most of the scripts used for this automation here</p>
<ul>
<li><a href="https://github.com/foodoh/automation_scripts">(https://github.com/foodoh/automation_scripts)</a> and</li>
<li><a href="https://github.com/foodoh/image_cleansing">(https://github.com/foodoh/image_cleansing)</a></li>
</ul>
<p>Stay tuned for more!</p>
Trying out Oculus Rift: Development kit 2
2016-03-13T00:00:00+00:00
https://www.tasdikrahman.com/2016/03/13/Oculus-Rift-Development-kit-two-review
<p>I woke up and found myself inside the elevator. A rather spooky one I would say. Questions like <strong>“Why was I inside an elevator?”</strong> would come to your mind. I am coming to that.</p>
<h3 id="prologue">Prologue</h3>
<p>I was all alone inside the dimly lit elevator. A sudden jerk and the next thing I know is that the lift is going up.</p>
<p>I look around and it seems all normal to me. Next, the lift stops at the 4th floor. The door opens and I look outside the elevator door. I was expecting to see somebody as I didn’t press any of the buttons. Door closes automatically and up we go again.</p>
<p>We stop again at the 10th floor. I again don’t remember myself pressing anything, but what the heck right. The door opens and the next thing I see makes me jump two steps closer to the back of the elevator. What was it you ask?</p>
<center><img src="/content/images/2016/3/elev.jpg" /></center>
<h3 id="did-you-say-a-tricyle">Did you say a tricyle?</h3>
<p>It was a <strong>tricycle</strong>! The ones which we all used to have when we were in playschool! I assured myself that it must have been left by some kid playing around and that he/she must be nearby the door. I try peeping out but all I see is utter darkness. Out of nowhere, a basketball is thrown inside the lift and I swear to god I shouted like anything!</p>
<p>The door closes again. The lift starts moving up and suddenly there is complete blackness inside. It takes me a moment to realize that there has been a powercut. I try looking for my cellphone in my pockets. But I can’t feel anything there.</p>
<p>After some restless moments, the power comes back. The lift starts moving up. I notice something really strange on the lift’s display. The numerics which display the floor on which the lift currently is, starts flickering. I don’t pay attention to it thinking that it might be due to some technical glitch. But what spookes me out is when the display decides to go haywire and display text which looks more of some foreign language to me!</p>
<p>Now I am frantically trying to press the buttons but all in vain. The lift stops and the door opens. I look up and see the same tricycle at the same place where it was on the floor beneath. Talk about coincidences when you don’t have any. Poor brain trying to calm you down I guess. Anyways.</p>
<h3 id="hello-is-anybody-there">Hello! Is anybody there?</h3>
<p>The door closes after a brief 3-4 seconds and off we go again. I am contemplating about how the heck did I land up inside this lift and while I am thinking all this, the door opens again and I freeze.</p>
<p>I look up expecting the same old tricycle but I see none which does little to relieve me. I move forward to leave the elevator and the next thing I know is a dark shadow landing right in front of me! I mean right on my freckin’ face.</p>
<p>As I compose myself, I see the man standing right in front me. A black suit, red tie. I watch the masked man as I take fumbled steps towards the back of the elevator.
I watch in horror, as the figure slowly grows in size and reaches a height in which his head literally touches the elevator top.</p>
<p>In a flick of a second, the masked man lungs at me and I fall down on the ground so as to dogde him.</p>
<hr />
<h3 id="and-then-i-took-off-the-oculus-rift-over-from-my-head">And then I took off the <strong>Oculus Rift</strong> over from my head!</h3>
<blockquote>
<p>It all happened so fast that I was left panting for more!</p>
</blockquote>
<p>Sorry for the long intro, but it would be shame if I did not explain the level of immersiveness of an Oculus with an experience. You have to try this thing out however you can and I cannot even begin to tell you how much I enjoyed the experience!</p>
<center><img src="/content/images/2016/3/elev_man.jpg" /></center>
<p>It’s been two week or so, since <a href="www.srmmilan.com/">Milan 2k16</a> (our college cultural fest) has passed by. And I thank the guys over at <strong>Nvidia</strong> (being one of the many great sponsers that we had) and the guys at <strong>TGN</strong> for bringing over super cool gadgets and gaming rigs over to our college.</p>
<p>Enjoyed the fest and all the events in it, but this was the cherry on top for me.</p>
<hr />
<p>Here’s me being dumbfounded by the next big thing after the internet!</p>
<center><img src="/content/images/2016/3/oculus_milan.jpg" /></center>
Making of space Shooter using pygame
2016-02-02T00:00:00+00:00
https://www.tasdikrahman.com/2016/02/02/Making-of-space-Shooter-using-pygame
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" />
<p>I procrastinated enough in writing this post so here it goes. <code class="language-plaintext highlighter-rouge">Pygame</code> treated me good. So good that I was able to create a decent enough <code class="language-plaintext highlighter-rouge">2-D</code> game in a day!</p>
<p>So here is my breakdown of it.</p>
<h2 id="creating-the-basic-rectangles">Creating the basic rectangles</h2>
<center><img src="http://i.imgur.com/50qgY67.jpg" /></center>
<p>I used the <a href="https://github.com/tasdikrahman/pygame-boilerplate">pygame-boilerplate</a> which I made in the process of making this game.</p>
<p>It’s nothing groundbreaking. Provides just a basic starting ground for you to base your pygame projects. Saves you some gruntwork.</p>
<h2 id="adding-the-enemy-sprites">Adding the enemy sprites</h2>
<center><img src="http://i.imgur.com/HorSt1T.jpg" /></center>
<p>At this point, the collisions needed to be added. When the player collided with a mob sprite, the game needs to end.</p>
<h2 id="adding-the-icons-for-all">Adding the icons for all</h2>
<center><img src="http://i.imgur.com/QV57Zqb.jpg" /></center>
<p>Adding the icons for the sprites was not that hard. I got the icons from <a href="http://opengameart.org/">opengameart.org/</a>, more particulary from the <a href="http://opengameart.org/content/space-shooter-redux">Space shooter content</a> pack from <a href="http://opengameart.org/users/kenney">@kenney</a>.</p>
<p>License for them is in <code class="language-plaintext highlighter-rouge">Public Domain</code>. This pack is a gem of a package. I mean you get all what you need in this pack!</p>
<p>The sound effects came next which included the explosions for the mob sprites as well as the player.</p>
<p>How about adding sound effects when shooting the missiles? Done deal!</p>
<h2 id="finishing-it-up">Finishing it up</h2>
<center><img src="http://i.imgur.com/1Zraayf.jpg" /></center>
<p>Done with the sound effects. The explosion animations. Was Left with adding things like high scores, player lives, health bar.</p>
<p>Something which I had not planned were powerups like shields and power ups. Dealing with Github feature requests anybody?</p>
<p>So this is what is what the main menu looks like</p>
<center><img src="http://i.imgur.com/3MzfmbT.jpg" /></center>
<p>In the end. I loved making this game a lot and I hope you make something much cooler than this. Do share it when you do.</p>
<p>Here’s the git repo if you are interested in taking a look at the source code.</p>
<hr />
<p><i class="fa fa-github-alt fa-2x"></i> <a href="https://github.com/tasdikrahman/spaceShooter">tasdikrahman/spaceShooter</a></p>
<h2 id="wanna-play">Wanna play?</h2>
<p>Have a nostalgic trip back to your childhood playing it! You can Download it for your preferred system.</p>
<p>Best part, it requires no installation! Just unzip it and you are good to go</p>
<table>
<thead>
<tr>
<th style="text-align: center"><i class="fa fa-linux fa-2x"></i></th>
<th style="text-align: center"><a href="https://github.com/tasdikrahman/spaceShooter/releases/download/v0.0.3/spaceShooter-v0.0.3_linux.zip">Download for linux based systems</a></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center"><i class="fa fa-windows fa-2x"></i></td>
<td style="text-align: center"><a href="https://github.com/tasdikrahman/spaceShooter/releases/download/v0.0.3/spaceShooter-v0.0.3_windows.zip">Download for windows based systems</a></td>
</tr>
</tbody>
</table>
<!-- <a class="btn btn-lg btn-success" href="https://github.com/tasdikrahman/spaceShooter/releases/download/v0.0.3/spaceShooter-v0.0.3_windows.zip">
<i class="fa fa-flag fa-2x pull-left"></i> Space Shooter - Windows <br>Version 0.0.3</a>
<a class="btn btn-lg btn-success" href="https://github.com/tasdikrahman/spaceShooter/releases/download/v0.0.3/spaceShooter-v0.0.3_linux.zip">
<i class="fa fa-flag fa-2x pull-left"></i> Space Shooter - linux <br>Version 0.0.3</a> -->
<p><strong>Support for MAC OS coming soon!</strong></p>
<hr />
<center><a href="https://github.com/tasdikrahman/spaceShooter"><img src="/content/images/2016/1/spaceShooter.gif" /></a></center>
<p>Happy coding!</p>
Say Hi to peewee
2016-01-29T00:00:00+00:00
https://www.tasdikrahman.com/2016/01/29/Using-peewee-as-an-ORM
<p>Once upon a time, when we had to interact with the databases. We had to write bare bones <code class="language-plaintext highlighter-rouge">SQL</code>(seequel if you may) or Structured Query language. A language which many common databases like</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">SQLite</code></li>
<li><code class="language-plaintext highlighter-rouge">MySQL</code></li>
<li><code class="language-plaintext highlighter-rouge">MariaDB</code> to name a few.</li>
</ul>
<p><code class="language-plaintext highlighter-rouge">SQL</code> is amazing but for day to day tasks it is pretty daunting and could be one of the ways in which you can shoot yourself squarely in the foot.</p>
<p>Now I am sure that you don’t want that to happen!</p>
<h2 id="enter-orm">Enter ORM</h2>
<p>ORM is an acronym for <code class="language-plaintext highlighter-rouge">Object Relational Mapping</code>. Okay, but what does an ORM do?</p>
<blockquote>
<p>It turns object in your code to rows in your database and vice versa.</p>
</blockquote>
<p>They do this by defining a model. A model represents a <code class="language-plaintext highlighter-rouge">table</code> in the database. The models are nothing but <code class="language-plaintext highlighter-rouge">classes</code> who have their attributes represent the <code class="language-plaintext highlighter-rouge">columns</code></p>
<p>I stumbled upon <a href="https://en.wikipedia.org/wiki/Peewee_ORM">Peewee</a>, a lightweight ORM tool for python.</p>
<table>
<tbody>
<tr>
<td><a href="https://github.com/coleifer/peewee">Github/peewee</a></td>
<td><a href="http://docs.peewee-orm.com/">Documentation</a></td>
</tr>
</tbody>
</table>
<p>Now you might ask, why <code class="language-plaintext highlighter-rouge">peewee</code> and not any other ORM like <code class="language-plaintext highlighter-rouge">SQLAlchemy</code> or <code class="language-plaintext highlighter-rouge">Storm</code> for the matter?</p>
<p>Well my reason for that would be the closeness to the <code class="language-plaintext highlighter-rouge">Django</code> style declaration models. This would be immensely helpful for anybody who is gonna be learning/working with <code class="language-plaintext highlighter-rouge">Django</code>. Plus it’s lightweight!</p>
<p>Here is a quick demo for how to use it.</p>
<h2 id="peewee-a-gentle-intro">Peewee, A gentle intro</h2>
<p>Let’s try to model a <code class="language-plaintext highlighter-rouge">Student</code> database</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">peewee</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">SqliteDatabase</span><span class="p">(</span><span class="s">'student.db'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Student</span><span class="p">(</span><span class="n">Model</span><span class="p">):</span>
<span class="n">username</span> <span class="o">=</span> <span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">255</span><span class="p">,</span> <span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">points</span> <span class="o">=</span> <span class="n">IntegerField</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">database</span> <span class="o">=</span> <span class="n">db</span></code></pre></figure>
<p>The model has been created.</p>
<p>Let’s enter some students to the database</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">students</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span><span class="s">'username'</span><span class="p">:</span> <span class="s">'tasdik'</span><span class="p">,</span>
<span class="s">'points'</span><span class="p">:</span> <span class="mi">200</span>
<span class="p">},</span>
<span class="p">{</span><span class="s">'username'</span><span class="p">:</span> <span class="s">'kellogs'</span><span class="p">,</span>
<span class="s">'points'</span><span class="p">:</span> <span class="mi">400</span>
<span class="p">},</span>
<span class="p">{</span><span class="s">'username'</span><span class="p">:</span> <span class="s">'john'</span><span class="p">,</span>
<span class="s">'points'</span><span class="p">:</span> <span class="mi">500</span>
<span class="p">},</span>
<span class="p">{</span><span class="s">'username'</span><span class="p">:</span> <span class="s">'doe'</span><span class="p">,</span>
<span class="s">'points'</span><span class="p">:</span> <span class="mi">600</span>
<span class="p">},</span>
<span class="p">{</span><span class="s">'username'</span><span class="p">:</span> <span class="s">'foo'</span><span class="p">,</span>
<span class="s">'points'</span><span class="p">:</span> <span class="mi">1000</span>
<span class="p">},</span>
<span class="p">]</span></code></pre></figure>
<p>How about we make the process of creating a seperate function for adding the students to the database?</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">add_student</span><span class="p">():</span>
<span class="k">for</span> <span class="n">student</span> <span class="ow">in</span> <span class="n">students</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">Student</span><span class="p">.</span><span class="n">create</span><span class="p">(</span>
<span class="n">username</span><span class="o">=</span><span class="n">student</span><span class="p">[</span><span class="s">'username'</span><span class="p">],</span>
<span class="n">points</span><span class="o">=</span><span class="n">student</span><span class="p">[</span><span class="s">'points'</span><span class="p">]</span>
<span class="p">)</span>
<span class="k">except</span> <span class="n">IntegrityError</span><span class="p">:</span>
<span class="n">student_record</span> <span class="o">=</span> <span class="n">Student</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">username</span><span class="o">=</span><span class="n">student</span><span class="p">[</span><span class="s">'username'</span><span class="p">])</span>
<span class="k">if</span> <span class="n">student</span><span class="p">[</span><span class="s">'points'</span><span class="p">]</span> <span class="o">!=</span> <span class="n">student_record</span><span class="p">.</span><span class="n">points</span><span class="p">:</span>
<span class="n">student_record</span><span class="p">.</span><span class="n">points</span> <span class="o">=</span> <span class="n">Student</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">points</span><span class="o">=</span><span class="n">student</span><span class="p">[</span><span class="s">'points'</span><span class="p">])</span>
<span class="n">student_record</span><span class="p">.</span><span class="n">save</span><span class="p">()</span></code></pre></figure>
<h2 id="checking-whether-there-is-anything-in-there">Checking whether there is anything in there</h2>
<p>Firing up the interpreter and running <code class="language-plaintext highlighter-rouge">sqlite3</code></p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="err">$</span> <span class="n">sqlite3</span> <span class="n">students</span><span class="p">.</span><span class="n">db</span>
<span class="c1">-- Loading resources from /home/tasdik/.sqliterc</span>
<span class="n">SQLite</span> <span class="k">version</span> <span class="mi">3</span><span class="p">.</span><span class="mi">8</span><span class="p">.</span><span class="mi">6</span> <span class="mi">2014</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">15</span> <span class="mi">11</span><span class="p">:</span><span class="mi">46</span><span class="p">:</span><span class="mi">33</span>
<span class="n">Enter</span> <span class="nv">".help"</span> <span class="k">for</span> <span class="k">usage</span> <span class="n">hints</span><span class="p">.</span>
<span class="n">sqlite</span><span class="o">></span> <span class="p">.</span><span class="n">tables</span>
<span class="n">student</span>
<span class="n">sqlite</span><span class="o">></span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">student</span><span class="p">;</span>
<span class="n">id</span> <span class="n">username</span> <span class="n">points</span>
<span class="c1">---------- ---------- ----------</span>
<span class="mi">1</span> <span class="n">tasdik</span> <span class="mi">200</span>
<span class="mi">2</span> <span class="n">kellogs</span> <span class="mi">400</span>
<span class="mi">3</span> <span class="n">john</span> <span class="mi">500</span>
<span class="mi">4</span> <span class="n">doe</span> <span class="mi">600</span>
<span class="mi">5</span> <span class="n">foo</span> <span class="mi">1000</span>
<span class="n">sqlite</span><span class="o">></span> <span class="p">.</span><span class="n">exit</span></code></pre></figure>
<p>Querying the database becomes as simple as a breeze too!</p>
<h2 id="the-whole-thing-put-together">The whole thing put together</h2>
<p>So there you go</p>
<script src="https://gist.github.com/99adec66eda0754d853c.js"> </script>
<h2 id="some-notes---if-you-may">Some notes - If you may:</h2>
<ul>
<li><code class="language-plaintext highlighter-rouge">model</code> - A code object that represents a database table</li>
<li><code class="language-plaintext highlighter-rouge">SqliteDatabase</code> - The class from Peewee that lets us connect to an SQLite database</li>
<li><code class="language-plaintext highlighter-rouge">Model</code> - The Peewee class that we extend to make a model</li>
<li><code class="language-plaintext highlighter-rouge">CharField</code> - A Peewee field that holds onto characters. It’s a varchar in SQL terms</li>
<li><code class="language-plaintext highlighter-rouge">max_length</code> - The maximum number of characters in a CharField</li>
<li><code class="language-plaintext highlighter-rouge">IntegerField</code> - A Peewee field that holds an integer</li>
<li><code class="language-plaintext highlighter-rouge">default</code> - A default value for the field if one isn’t provided</li>
<li><code class="language-plaintext highlighter-rouge">unique</code> - Whether the value in the field can be repeated in the table</li>
<li><code class="language-plaintext highlighter-rouge">.connect()</code> - A database method that connects to the database</li>
<li><code class="language-plaintext highlighter-rouge">.create_tables()</code> - A database method to create the tables for the specified models.</li>
<li><code class="language-plaintext highlighter-rouge">safe</code> - Whether or not to throw errors if the table(s) you’re attempting to create already exist</li>
<li><code class="language-plaintext highlighter-rouge">.create()</code> - creates a new instance all at once</li>
<li><code class="language-plaintext highlighter-rouge">.select()</code> - finds records in a table</li>
<li><code class="language-plaintext highlighter-rouge">.save()</code> - updates an existing row in the database</li>
<li><code class="language-plaintext highlighter-rouge">.get()</code> - finds a single record in a table</li>
<li><code class="language-plaintext highlighter-rouge">.delete_instance()</code> - deletes a single record from the table</li>
<li><code class="language-plaintext highlighter-rouge">.order_by()</code> - specify how to sort the records</li>
<li><code class="language-plaintext highlighter-rouge">.update()</code> - also something we didn’t use. Offers a way to update a record without <code class="language-plaintext highlighter-rouge">.get()</code> and <code class="language-plaintext highlighter-rouge">.save()</code></li>
<li><code class="language-plaintext highlighter-rouge">.where()</code> - method that lets us filter our .select() results</li>
<li><code class="language-plaintext highlighter-rouge">.contains()</code> - method that specifies the input should be inside the specified field</li>
</ul>
<p>Example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Student.update(points=student['points']).where(Student.username == student['username']).execute()
</code></pre></div></div>
<h2 id="references">References:</h2>
<ul>
<li><a href="http://peewee.readthedocs.org/en/latest/peewee/querying.html">Peewee query methods</a></li>
<li><a href="http://stackoverflow.com/questions/53428/what-are-some-good-python-orm-solutions">Stackoverflow: List of Python ORM tools</a></li>
<li><a href="https://en.wikipedia.org/wiki/List_of_object-relational_mapping_software">Wiki: ORM software lists</a></li>
<li><a href="https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior">strftime options</a></li>
</ul>
Getting started with Pygame
2016-01-17T00:00:00+00:00
https://www.tasdikrahman.com/2016/01/17/Pygame-Introduction
<h2 id="pygame-intro">Pygame intro</h2>
<p>As with every other kid out there, I spent long hours sitting in front of the computer playing Games like <a href="https://en.wikipedia.org/wiki/Super_Mario">Super Mario</a>, <a href="https://en.wikipedia.org/wiki/Dangerous_Dave">Dangerous Dave</a> and the likes. So when I got to know about <a href="www.pygame.org/">Pygame</a>. I was really getting the itch on creating something in the lines of these games.</p>
<p>Of cource, we have better game engines written in other languages like <code class="language-plaintext highlighter-rouge">C++</code>, but since I liked <code class="language-plaintext highlighter-rouge">python</code>. So I mean what the heck right?</p>
<p>Lets Get Started then shall we</p>
<h2 id="installation">Installation</h2>
<h3 id="ubuntudebian">Ubuntu/Debian</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>python-pygame</code></pre></figure>
<h3 id="os-x">OS X</h3>
<p>If you DON’T have <code class="language-plaintext highlighter-rouge">homebrew</code> installed, then</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>ruby <span class="nt">-e</span> <span class="s2">"</span><span class="si">$(</span>curl <span class="nt">-fsSL</span> https://raw.githubusercontent.com/Homebrew/install/master/install<span class="si">)</span><span class="s2">"</span></code></pre></figure>
<p>If you have it installed, then</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>brew <span class="nb">install </span>caskroom/cask/brew-cask
<span class="nv">$ </span>brew cask <span class="nb">install </span>xquartz
<span class="nv">$ </span>brew <span class="nb">install </span>python3
<span class="nv">$ </span>brew <span class="nb">install </span>python
<span class="nv">$ </span>brew linkapps python3
<span class="nv">$ </span>brew linkapps python
<span class="nv">$ </span>brew <span class="nb">install </span>git
<span class="nv">$ </span>brew <span class="nb">install </span>sdl sdl_image sdl_ttf portmidi libogg libvorbis
<span class="nv">$ </span>brew <span class="nb">install </span>sdl_mixer <span class="nt">--with-libvorbis</span>
<span class="nv">$ </span>brew tap homebrew/headonly
<span class="nv">$ </span>brew <span class="nb">install </span>smpeg
<span class="nv">$ </span>brew <span class="nb">install </span>mercurial
<span class="nv">$ </span>pip3 <span class="nb">install </span>hg+http://bitbucket.org/pygame/pygame</code></pre></figure>
<h2 id="see-if-it-works">See if it works?</h2>
<p>Open the terminal</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>python
Python 2.7.8 <span class="o">(</span>default, Jun 18 2015, 18:54:19<span class="o">)</span>
<span class="o">[</span>GCC 4.9.1] on linux2
Type <span class="s2">"help"</span>, <span class="s2">"copyright"</span>, <span class="s2">"credits"</span> or <span class="s2">"license"</span> <span class="k">for </span>more information.
<span class="o">>>></span> import pygame
<span class="o">>>></span></code></pre></figure>
<p>If there is no Error on that, then you are good to go.</p>
<h2 id="a-simple-pygame-boilerplate">A simple Pygame Boilerplate</h2>
<p>I made a dead simple <code class="language-plaintext highlighter-rouge">Pygame</code> boilerplate for creating a base for your <code class="language-plaintext highlighter-rouge">pygame</code> programs.</p>
<p>Here is the Github link for you.</p>
<ul>
<li>link : <a href="https://github.com/tasdikrahman/pygame-boilerplate">tasdikrahman/pygame-boilerplate</a></li>
</ul>
<p>A sneak peak</p>
<center><img src="http://i.imgur.com/p7PxMZl.jpg" /></center>
<h2 id="what-can-you-do-with-this">What can you do with this?</h2>
<p>I created <a href="https://github.com/tasdikrahman/spaceShooter">Space Shooter</a> using this boilerplate. Here’s a demo screen for you</p>
<center><img src="http://i.imgur.com/I5mTBFB.png" /></center>
<h2 id="references">References</h2>
<ul>
<li><a href="http://askubuntu.com/questions/399824/how-to-install-pygame">http://askubuntu.com/questions/399824/how-to-install-pygame</a></li>
<li><a href="https://inventwithpython.com/pygame/chapter1.html">https://inventwithpython.com/pygame/chapter1.html</a></li>
<li><a href="http://kidscancode.org/blog/2015/09/pygame_install/">http://kidscancode.org/blog/2015/09/pygame_install/</a></li>
</ul>
Unicode strings in python, a gentle intro
2015-12-08T00:00:00+00:00
https://www.tasdikrahman.com/2015/12/08/Working-with-unicode
<h2 id="summary">Summary</h2>
<p>In this post I will try to explain how to handle them in <code class="language-plaintext highlighter-rouge">python 2 and 3</code>.</p>
<p>I had long undermined the way I handled strings in my projects, but I could feel the gravity of handling <code class="language-plaintext highlighter-rouge">strings</code> properly when I was working on <a href="https://github.com/tasdikrahman/vocabulary/">vocabulary</a>, a side project of mine.</p>
<p>There was this one feature in it where the <code class="language-plaintext highlighter-rouge">module</code> had to return the <code class="language-plaintext highlighter-rouge">pronunciation</code> for a given word. Well I wrote the logic to parse the content and all the stuff. I had it all figured out, but then I was facing <a href="https://github.com/tasdikrahman/vocabulary/#known-issues">this issue</a>.</p>
<p>Let’s start shall we?</p>
<h2 id="ascii-strings">ASCII strings</h2>
<p>So let’s start with the <code class="language-plaintext highlighter-rouge">ASCII</code> strings, Have a look at <a href="../../../../content/unicode/hi.txt">hi.txt</a></p>
<p>Let’s see what does it hold</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik at Acer <span class="k">in</span> ~/unicode
<span class="nv">$ </span><span class="nb">cat </span>hi.txt
hi</code></pre></figure>
<p>Nice and easy. It contains, two characters <code class="language-plaintext highlighter-rouge">h</code> and <code class="language-plaintext highlighter-rouge">i</code></p>
<p>Size ?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik at Acer <span class="k">in</span> ~/unicode
<span class="nv">$ </span><span class="nb">du</span> <span class="nt">-a</span> <span class="nt">-b</span> hi.txt
2 hi.txt</code></pre></figure>
<p>This means that the file is of <code class="language-plaintext highlighter-rouge">2 bytes</code>. Now what do these <code class="language-plaintext highlighter-rouge">2 bytes</code> hold inside them? Let’s do a <code class="language-plaintext highlighter-rouge">hexdump</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik at Acer <span class="k">in</span> ~/unicode
<span class="nv">$ </span>hexdump hi.txt
0000000 6968
0000002</code></pre></figure>
<p>If you look over to the <a href="http://www.asciitable.com/">ASCII table</a> and look out for the hex representations, you will see that the letter <code class="language-plaintext highlighter-rouge">h</code> is represented by <code class="language-plaintext highlighter-rouge">68</code> and <code class="language-plaintext highlighter-rouge">i</code> is represented by <code class="language-plaintext highlighter-rouge">69</code></p>
<p>Let’s see how <code class="language-plaintext highlighter-rouge">python2</code> handles this. Firing up the <code class="language-plaintext highlighter-rouge">interpreter</code></p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'hi.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="p">...</span> <span class="n">content</span> <span class="o">=</span> <span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">()</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">content</span>
<span class="s">'hi'</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="o"><</span><span class="nb">type</span> <span class="s">'str'</span><span class="o">></span>
<span class="o">>>></span> </code></pre></figure>
<p>Now I probably should reiterate the fact that</p>
<blockquote>
<p>Every character in a string is a single byte</p>
</blockquote>
<p>And that the ASCII table translates each byte value to a unique character. the file contains an <code class="language-plaintext highlighter-rouge">ASCII</code> string of exactly two characters. So it does makes sense. Let’s dig a little further.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="nb">len</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="mi">2</span>
<span class="o">>>></span> <span class="n">content</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="s">'h'</span>
<span class="o">>>></span> <span class="n">content</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="s">'i'</span>
<span class="o">>>></span></code></pre></figure>
<p>So this confirms that the <code class="language-plaintext highlighter-rouge">x[0]</code> contains <code class="language-plaintext highlighter-rouge">h</code> and <code class="language-plaintext highlighter-rouge">x[1]</code> contains <code class="language-plaintext highlighter-rouge">i</code></p>
<h2 id="enter-unicode">Enter Unicode</h2>
<p>So how many characters does the <code class="language-plaintext highlighter-rouge">ASCII</code> representation able to represent? Doing the math, 256(<code class="language-plaintext highlighter-rouge">2^8</code>) would be the maximum number of characters that the <code class="language-plaintext highlighter-rouge">ASCII</code> table can represent. Just giving a heads up here, <code class="language-plaintext highlighter-rouge">Chinese</code> has a lot more than <code class="language-plaintext highlighter-rouge">256</code> characters. So how would you handle <code class="language-plaintext highlighter-rouge">chinese</code> as well as the characters on your keyboard?</p>
<p>Have a look at <a href="../../../../content/unicode/chinese.txt">chinese.txt</a></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik at Acer <span class="k">in</span> ~/unicode
<span class="nv">$ </span><span class="nb">cat </span>chinese.txt
hi猫</code></pre></figure>
<p>So it contains three character namely <code class="language-plaintext highlighter-rouge">h</code>, <code class="language-plaintext highlighter-rouge">i</code> and <code class="language-plaintext highlighter-rouge">猫</code>. Size?</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik at Acer <span class="k">in</span> ~/unicode
<span class="nv">$ </span><span class="nb">du</span> <span class="nt">-a</span> <span class="nt">-b</span> chinese.txt
5 chinese.txt</code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">5 bytes</code>. Let’s see what does each byte contain</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik at Acer <span class="k">in</span> ~/unicode
<span class="nv">$ </span>hexdump chinese.txt
0000000 68 69 e7 8c ab
0000005</code></pre></figure>
<p>The relevant thing to note here are the five <code class="language-plaintext highlighter-rouge">hexadecimal</code> numbers <code class="language-plaintext highlighter-rouge">69</code>, <code class="language-plaintext highlighter-rouge">68</code>, <code class="language-plaintext highlighter-rouge">e7</code>, <code class="language-plaintext highlighter-rouge">8c</code> and <code class="language-plaintext highlighter-rouge">ab</code></p>
<p>So five numbers, 5 bytes. Good so far? Now how do we interpret these numbers? We will have a look at the <a href="https://en.wikipedia.org/wiki/UTF-8">Unicode UTF-8</a> table.</p>
<p>In this table, <code class="language-plaintext highlighter-rouge">68</code> is the character <code class="language-plaintext highlighter-rouge">h</code>, <code class="language-plaintext highlighter-rouge">69</code> is the character <code class="language-plaintext highlighter-rouge">i</code>, and the three-byte sequence <code class="language-plaintext highlighter-rouge">e7</code>, <code class="language-plaintext highlighter-rouge">8c</code>, <code class="language-plaintext highlighter-rouge">ab</code> is the character <code class="language-plaintext highlighter-rouge">猫</code>. To recap, <code class="language-plaintext highlighter-rouge">h</code> is one byte, <code class="language-plaintext highlighter-rouge">i</code> is one byte, but <code class="language-plaintext highlighter-rouge">猫</code> is three bytes.</p>
<p>A point to note here is that, the Unicode UTF-8 table is a superset of the ASCII table, so that’s the reason <code class="language-plaintext highlighter-rouge">h</code> and <code class="language-plaintext highlighter-rouge">i</code> are represented by the same characters in both.</p>
<h2 id="handling-unicode-strings-in-python2">Handling unicode strings in <code class="language-plaintext highlighter-rouge">python2</code></h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'chinese.txt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="p">...</span> <span class="n">content</span> <span class="o">=</span> <span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">()</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">content</span>
<span class="s">'hi</span><span class="se">\xe7\x8c\xab</span><span class="s">'</span>
<span class="o">>>></span> <span class="nb">len</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="mi">5</span>
<span class="o">>>></span></code></pre></figure>
<p>What was all that? <code class="language-plaintext highlighter-rouge">h</code> and <code class="language-plaintext highlighter-rouge">i</code> are represented just fine but when it comes to the chinese character, it shows me hexdecimal numbers. And how does it return me <code class="language-plaintext highlighter-rouge">5</code> as the string lenght, when we know perfectly well that there are just <code class="language-plaintext highlighter-rouge">3</code> characters in that file?</p>
<p>It turns out that the python <code class="language-plaintext highlighter-rouge">str</code> doesn’t store a <code class="language-plaintext highlighter-rouge">string</code> but a stream of <code class="language-plaintext highlighter-rouge">bytes</code> in it. Digging further.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">>>></span> x[0]
<span class="s1">'h'</span>
<span class="o">>>></span> x[1]
<span class="s1">'i'</span>
<span class="o">>>></span> x[2]
<span class="s1">'\xe7'</span>
<span class="o">>>></span> x[3]
<span class="s1">'\x8c'</span>
<span class="o">>>></span> x[4]
<span class="s1">'\xab'</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">hi</code> is returned prefectly fine as those are ASCII characters, but when it comes to the chinese character, it is represented by UTF-8 unicode. But since <code class="language-plaintext highlighter-rouge">str</code> object in <code class="language-plaintext highlighter-rouge">python2</code> just stores a sequece of bytes, it has no way of deciding to group these 3 characters to represent the chinese character. So we see them as the hexadecimal numbers.</p>
<p>So how should we deal with this.</p>
<h2 id="decode-to-the-rescue"><code class="language-plaintext highlighter-rouge">decode()</code> to the rescue</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">>>></span> utf_content <span class="o">=</span> content.decode<span class="o">(</span><span class="s1">'utf-8'</span><span class="o">)</span>
<span class="o">>>></span> utf_content
u<span class="s1">'hi\u732b'</span>
<span class="o">>>></span> <span class="nb">type</span><span class="o">(</span>utf_content<span class="o">)</span>
<<span class="nb">type</span> <span class="s1">'unicode'</span><span class="o">></span>
<span class="o">>>></span> len<span class="o">(</span>utf_content<span class="o">)</span>
3
<span class="o">>>></span> utf_content[0]
u<span class="s1">'h'</span>
<span class="o">>>></span> utf_content[1]
u<span class="s1">'i'</span>
<span class="o">>>></span> utf_content[2]
u<span class="s1">'\u732b'</span>
<span class="o">>>></span></code></pre></figure>
<p>So the <code class="language-plaintext highlighter-rouge">decode()</code> tells python to convert the string <code class="language-plaintext highlighter-rouge">content</code> into a <code class="language-plaintext highlighter-rouge">UTF-8</code> string. I know, the name is confusing as hell. But let’s leave that for another day.</p>
<p>I we call the <code class="language-plaintext highlighter-rouge">print</code> statement now. Let’s see what we get</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">>>></span> print utf_content
hi猫
<span class="o">>>></span></code></pre></figure>
<p>So there you go.</p>
<h2 id="word-of-caution">Word of caution</h2>
<p>Weird things happen in <code class="language-plaintext highlighter-rouge">python2</code> if you think that <code class="language-plaintext highlighter-rouge">str</code> is a <code class="language-plaintext highlighter-rouge">string</code>. To be safe, convert the <code class="language-plaintext highlighter-rouge">str</code> object to <code class="language-plaintext highlighter-rouge">utf-8</code> format immediately by doing a <code class="language-plaintext highlighter-rouge">decode('utf-8')</code>. Then work with your <code class="language-plaintext highlighter-rouge">unicode</code> object and not the <code class="language-plaintext highlighter-rouge">str</code> or else you will some real pain handling the issues. Like I had in <a href="https://github.com/tasdikrahman/vocabulary#known-issues">vocabulary</a></p>
<blockquote>
<p>In python2, a unicode object type represents real strings whereas the str object is a sequece of bytes.</p>
</blockquote>
<p>So when you are done precessing your <code class="language-plaintext highlighter-rouge">unicode</code> object and now you want to write it down to a file or a database. First convert it back to a sequence of <code class="language-plaintext highlighter-rouge">bytes</code> (<code class="language-plaintext highlighter-rouge">str</code> object) using the <code class="language-plaintext highlighter-rouge">encode()</code> method.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">str_content</span> <span class="o">=</span> <span class="n">utf_content</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="n">str_content</span><span class="p">)</span>
<span class="o"><</span><span class="nb">type</span> <span class="s">'str'</span><span class="o">></span>
<span class="o">>>></span> <span class="n">str_content</span>
<span class="s">'hi</span><span class="se">\xe7\x8c\xab</span><span class="s">'</span>
<span class="o">>>></span> <span class="n">content</span> <span class="o">==</span> <span class="n">str_content</span>
<span class="bp">True</span>
<span class="o">>>></span> </code></pre></figure>
<p>Now you will be able to write this content to a file or database as directly doing so with a <code class="language-plaintext highlighter-rouge">unicode</code> object would have given you some wierd errors.</p>
<p>Okay, okay. I will show that to you</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'myfile.txt'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="p">...</span> <span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">utf_content</span><span class="p">)</span>
<span class="p">...</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">2</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="nb">UnicodeEncodeError</span><span class="p">:</span> <span class="s">'ascii'</span> <span class="n">codec</span> <span class="n">can</span><span class="s">'t encode character u'</span>\<span class="n">u732b</span><span class="s">' in position 2: ordinal not in range(128)
>>></span></code></pre></figure>
<p>Now doing the same with the <code class="language-plaintext highlighter-rouge">str</code> object</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">>>></span> with open<span class="o">(</span><span class="s1">'myfile.txt'</span>, <span class="s1">'w'</span><span class="o">)</span> as f:
... f.write<span class="o">(</span>str_content<span class="o">)</span>
...
<span class="o">>>></span></code></pre></figure>
<h2 id="handling-unicode-strings-in-python3">Handling unicode strings in <code class="language-plaintext highlighter-rouge">python3</code></h2>
<p>Python3 makes handling of unicode strings easy.</p>
<p>One of the significant changes being that, <code class="language-plaintext highlighter-rouge">str</code> now stores unicode <code class="language-plaintext highlighter-rouge">strings</code> and not a sequence of <code class="language-plaintext highlighter-rouge">bytes</code></p>
<p>Let’s see how it handles the <a href="../../../../content/unicode/chinese.txt">chinese.txt</a> file</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik at Acer <span class="k">in</span> ~/unicode
<span class="nv">$ </span>python3
Python 3.4.2 <span class="o">(</span>default, Jun 19 2015, 11:34:49<span class="o">)</span>
<span class="o">[</span>GCC 4.9.1] on linux
Type <span class="s2">"help"</span>, <span class="s2">"copyright"</span>, <span class="s2">"credits"</span> or <span class="s2">"license"</span> <span class="k">for </span>more information.
<span class="o">>>></span> with open<span class="o">(</span><span class="s1">'chinese.txt'</span><span class="o">)</span> as f:
... content <span class="o">=</span> f.read<span class="o">()</span>
...
<span class="o">>>></span> <span class="nb">type</span><span class="o">(</span>content<span class="o">)</span>
<class <span class="s1">'str'</span><span class="o">></span>
<span class="o">>>></span> len<span class="o">(</span>content<span class="o">)</span>
3
<span class="o">>>></span> content[0]
<span class="s1">'h'</span>
<span class="o">>>></span> content[1]
<span class="s1">'i'</span>
<span class="o">>>></span> content[2]
<span class="s1">'猫'</span></code></pre></figure>
<p>So everything works out of the box(Going with the Batteries included philosophy of <code class="language-plaintext highlighter-rouge">python</code>).</p>
<p>Now what if I wanted to interpret the contents of it as <code class="language-plaintext highlighter-rouge">bytes</code>.</p>
<p>You can do so by passing the argument <code class="language-plaintext highlighter-rouge">rb</code> when opening the file</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'chinese.txt'</span><span class="p">,</span> <span class="s">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="p">...</span> <span class="n">content</span> <span class="o">=</span> <span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">()</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">bytes</span><span class="s">'>
>>> content
b'</span><span class="n">hi</span>\<span class="n">xe7</span>\<span class="n">x8c</span>\<span class="n">xab</span><span class="s">'</span></code></pre></figure>
<p>So now you have got the default behaviour of <code class="language-plaintext highlighter-rouge">python2</code>.</p>
<p>Converting it into <code class="language-plaintext highlighter-rouge">utf-8</code></p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">content</span><span class="p">.</span><span class="n">decode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span>
<span class="s">'hi猫'</span></code></pre></figure>
<p>So to sum it up</p>
<blockquote>
<p>In <code class="language-plaintext highlighter-rouge">python3</code>, <code class="language-plaintext highlighter-rouge">str</code> represents <code class="language-plaintext highlighter-rouge">unicode</code> string while the <code class="language-plaintext highlighter-rouge">bytes</code> type represent the sequence of <code class="language-plaintext highlighter-rouge">bytes</code></p>
</blockquote>
<p>For further reading, I would really, really suggest you have a look on the content written by these guys</p>
<ul>
<li><a href="http://www.joelonsoftware.com/articles/Unicode.html">Joel Spolsky</a></li>
<li><a href="http://nedbatchelder.com/text/unipain.html">Ned Batchelder</a></li>
<li><a href="http://pgbovine.net/unicode-python.htm">Philip Guo</a></li>
</ul>
<p>on this topic</p>
Submitting python package to pypi
2015-11-10T00:00:00+00:00
https://www.tasdikrahman.com/2015/11/10/submitting-package-to-pypi
<p>Recently I had written a <a href="https://github.com/tasdikrahman/pyzipcode-cli/">thin wrapper around getziptastic’s API</a> and I wanted that to be availble as a <a href="pypi.python.org/pypi">pypi package</a>.</p>
<h2 id="what-is-pypi">What is <code class="language-plaintext highlighter-rouge">PyPI</code>?</h2>
<p>From the official website:</p>
<h4 id="pypi--the-python-package-index"><code class="language-plaintext highlighter-rouge">PyPI</code> — the Python Package Index</h4>
<p>The Python Package Index is a repository of software for the Python programming language.
Written something cool? Want others to be able to install it with easy_install or pip? Put your code on PyPI. It’s a big list of python packages that you absolutely must submit your package to for it to be easily one-line installable.</p>
<p>The good news is that submitting to PyPI is simple in theory: just sign up and upload your code, all for free. The bad news is that in practice it’s a little bit more complicated than that. The other good news is that I’ve written this guide, and that if you’re stuck, you can always refer to the official documentation.</p>
<p>Create your accounts</p>
<p>On <a href="http://pypi.python.org/pypi?%3Aaction=register_form">PyPI Live</a> and also on <a href="http://testpypi.python.org/pypi?%3Aaction=register_form">PyPI Test</a>. You must create an account in order to be able to upload your code. I recommend using the same email/password for both accounts, just to make your life easier when it comes time to push.</p>
<h2 id="create-a-pypirc-configuration-file">Create a <code class="language-plaintext highlighter-rouge">.pypirc</code> configuration file</h2>
<p>This file holds your information for authenticating with PyPI, both the live and the test versions. This file should be placed in your home directory. So do a</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>nano ~/.pypirc</code></pre></figure>
<p>Where you can use your favorite text editor in place of <code class="language-plaintext highlighter-rouge">nano</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>distutils] <span class="c"># this tells distutils what package indexes you can push to</span>
index-servers <span class="o">=</span>
pypi
pypitest
<span class="o">[</span>pypi]
repository: https://pypi.python.org/pypi
username: user_name
password: your_password
<span class="o">[</span>pypitest]
repository: https://testpypi.python.org/pypi
username: user_name
password: your_password
<span class="o">[</span>server-login]
repository: https://testpypi.python.org/pypi
username: user_name
password: your_password</code></pre></figure>
<p>This is just to make your life easier, so that when it comes time to upload you don’t have to type/remember your username and password.</p>
<h2 id="prepare-your-package">Prepare your package</h2>
<p>Every package on PyPI needs to have a file called <code class="language-plaintext highlighter-rouge">setup.py</code> at the root of the directory. If your’e using a markdown-formatted read me file you’ll also need a <code class="language-plaintext highlighter-rouge">setup.cfg</code> file. Also, you’ll want a <code class="language-plaintext highlighter-rouge">LICENSE.txt</code> file describing what can be done with your code. So if I’ve been working on a library called mypackage, my directory structure would look like this:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">.</span>
├── LICENSE.txt
├── pyzipcode-cli
│ ├── countries.json
│ ├── __init__.py
│ └── pyzipcode-cli.py
├── README.md
├── requirements.txt
├── setup.cfg
├── setup.py
└── usage.gif</code></pre></figure>
<p>Here’s a breakdown of what goes in which file:</p>
<h4 id="setuppy"><code class="language-plaintext highlighter-rouge">setup.py</code></h4>
<p>This is metadata about your library.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/usr/bin/env python</span>
try:
import os
from setuptools import setup, find_packages
except ImportError:
from distutils.core import setup
setup<span class="o">(</span>
name <span class="o">=</span> <span class="s1">'pyzipcode-cli'</span>,
version <span class="o">=</span> <span class="s1">'0.0.12'</span>,
author <span class="o">=</span> <span class="s1">'Tasdik Rahman'</span>,
author_email <span class="o">=</span> <span class="s1">'tasdikrahman@zoho.com'</span>,
<span class="c"># packages = ['pyzipcode_cli'], </span>
description <span class="o">=</span> <span class="s2">"a thin wrapper around getziptastic's API v2"</span>,
url <span class="o">=</span> <span class="s1">'https://github.com/tasdikrahman/pyzipcode-cli'</span>,
license <span class="o">=</span> <span class="s1">'MIT'</span>,
install_requires <span class="o">=</span> <span class="o">[</span>
<span class="s2">"docopt==0.6.1"</span>,
<span class="s2">"requests==2.8.1"</span>
<span class="o">]</span>,
<span class="c">### adding package data to it </span>
<span class="nv">packages</span><span class="o">=</span>find_packages<span class="o">(</span><span class="nv">exclude</span><span class="o">=[</span><span class="s1">'contrib'</span>, <span class="s1">'docs'</span>, <span class="s1">'tests'</span><span class="o">])</span>,
<span class="nv">package_data</span><span class="o">={</span>
<span class="s1">'pyzipcode_cli'</span>: <span class="o">[</span><span class="s1">'*.json'</span><span class="o">]</span>,
<span class="o">}</span>,
<span class="c">###</span>
download_url <span class="o">=</span> <span class="s1">'https://github.com/tasdikrahman/pyzipcode-cli/tarball/0.0.12'</span>,
classifiers <span class="o">=</span> <span class="o">[</span>
<span class="s1">'Intended Audience :: Developers'</span>,
<span class="s1">'Topic :: Software Development :: Build Tools'</span>,
<span class="s1">'License :: OSI Approved :: MIT License'</span>,
<span class="c"># Specify the Python versions you support here. In particular, ensure</span>
<span class="c"># that you indicate whether you support Python 2, Python 3 or both.</span>
<span class="s1">'Programming Language :: Python :: 2.7'</span>,
<span class="s1">'Programming Language :: Python :: 3.4'</span>,
<span class="o">]</span>,
keywords <span class="o">=</span> <span class="o">[</span><span class="s1">'api'</span>, <span class="s1">'geo-location'</span>, <span class="s1">'zipcode'</span>,<span class="s1">'devtools'</span>, <span class="s1">'Development'</span>, <span class="s1">'ziptastic'</span><span class="o">]</span>,
entry_points <span class="o">=</span> <span class="o">{</span>
<span class="s1">'console_scripts'</span>: <span class="o">[</span>
<span class="s1">'pyzipcode = pyzipcode_cli.core:main'</span>
<span class="o">]</span>,
<span class="o">}</span>
<span class="o">)</span></code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">download_url</code> is a link to a hosted file with your repository’s code. Github will host this for you, but only if you create a git tag.</p>
<p>In your repository, type:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>git tag 0.1 <span class="nt">-m</span> <span class="s2">"Adds a tag so that we can put this on PyPI."</span></code></pre></figure>
<p>Then, type <code class="language-plaintext highlighter-rouge">git tag</code> to show a list of tags — you should see 0.1 in the list.</p>
<p>Type</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git push <span class="nt">--tags</span> origin master</code></pre></figure>
<p>to update your code on Github with the latest tag information. Github creates tarballs for download at <code class="language-plaintext highlighter-rouge">https://github.com/{username}/{module_name}/tarball/{tag}</code>.</p>
<h4 id="setupcfg"><code class="language-plaintext highlighter-rouge">setup.cfg</code></h4>
<p>This tells PyPI where your README file is.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>metadata]
description-file <span class="o">=</span> README.md</code></pre></figure>
<p>This is necessary if you’re using a markdown readme file. At upload time, you may still get some errors about the lack of a readme — don’t worry about it. If you don’t have to use a markdown <code class="language-plaintext highlighter-rouge">README</code> file, I would recommend using reStructuredText (REST) instead.</p>
<h4 id="licensetxt"><code class="language-plaintext highlighter-rouge">LICENSE.txt</code></h4>
<p>This file will contain whichver license you want your code to have. I tend to use the <a href="prodicus.mit-license.org">MIT license</a>.</p>
<h2 id="upload-your-package-to-pypi-test">Upload your package to PyPI Test</h2>
<p>Run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>python setup.py register <span class="nt">-r</span> pypitest</code></pre></figure>
<p>This will attempt to register your package against PyPI’s test server, just to make sure you’ve set up everything correctly.</p>
<p>Then, run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>python setup.py sdist upload <span class="nt">-r</span> pypitest</code></pre></figure>
<p>You should get no errors, and should also now be able to see your library in the test PyPI repository.</p>
<h2 id="upload-to-pypi-live">Upload to PyPI Live</h2>
<p>Once you’ve successfully uploaded to PyPI Test, perform the same steps but point to the live PyPI server instead. To register, run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>python setup.py register <span class="nt">-r</span> pypi</code></pre></figure>
<p>Then, run:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>python setup.py sdist upload <span class="nt">-r</span> pypi</code></pre></figure>
<p>and you’re done!</p>
<h2 id="some-shameless-promotion">Some shameless promotion</h2>
<p>If you want to try my package out here is the</p>
<ul>
<li><a href="https://github.com/tasdikrahman/pyzipcode-cli/">github link to the repo</a></li>
<li><a href="https://pypi.python.org/pypi/pyzipcode-cli/">https://pypi.python.org/pypi/pyzipcode-cli/</a></li>
</ul>
<p>My cool looking badge :D</p>
<p><a href="https://badge.fury.io/py/pyzipcode-cli"><img src="https://badge.fury.io/py/pyzipcode-cli.svg" alt="PyPI version" /></a></p>
<h2 id="references">References:</h2>
<ul>
<li><a href="http://peterdowns.com/posts/first-time-with-pypi.html">http://peterdowns.com/posts/first-time-with-pypi.html</a></li>
<li><a href="https://the-hitchhikers-guide-to-packaging.readthedocs.org/en/latest/quickstart.html#lay-out-your-project">https://the-hitchhikers-guide-to-packaging.readthedocs.org/en/latest/quickstart.html#lay-out-your-project</a></li>
<li><a href="http://stackoverflow.com/questions/3658084/how-to-send-a-package-to-pypi?rq=1">http://stackoverflow.com/questions/3658084/how-to-send-a-package-to-pypi?rq=1</a></li>
<li><a href="http://packages.python.org/an_example_pypi_project/">http://packages.python.org/an_example_pypi_project/</a></li>
<li><a href="https://wiki.python.org/moin/CheeseShopTutorial">https://wiki.python.org/moin/CheeseShopTutorial</a></li>
<li><a href="http://docs.python.org/distutils/index.html">http://docs.python.org/distutils/index.html</a></li>
<li><a href="http://zetcode.com/articles/packageinpython/">http://zetcode.com/articles/packageinpython/</a></li>
<li><a href="http://stackoverflow.com/questions/18787036/difference-between-entry-points-console-scripts-and-scripts-in-setup-py">http://stackoverflow.com/questions/18787036/difference-between-entry-points-console-scripts-and-scripts-in-setup-py</a></li>
<li><a href="http://stackoverflow.com/questions/23324353/pros-and-cons-of-script-vs-entry-point-in-python-command-line-scripts?lq=1">http://stackoverflow.com/questions/23324353/pros-and-cons-of-script-vs-entry-point-in-python-command-line-scripts?lq=1</a></li>
</ul>
Creating a gif of the current window
2015-11-07T00:00:00+00:00
https://www.tasdikrahman.com/2015/11/07/creating-gif-from-screencast
<h2 id="backdrop">BackDrop:</h2>
<p>Everybody at some time of their life as netizen would have seen something like this</p>
<p><img src="http://i.stack.imgur.com/0B664.gif" alt="http://i.stack.imgur.com/0B664.gif" /></p>
<p>How about we create one?</p>
<p>Well recently when I was building a Calculator app, I wanted a <code class="language-plaintext highlighter-rouge">gif</code> image to be there in the <code class="language-plaintext highlighter-rouge">README.md</code> so as to show the usage of the app.</p>
<blockquote>
<p>I wrote a <a href="http://www.tasdikrahman.com/2015/11/06/Building-a-calculator/">How hard can Building a calclator be right?</a> for the same some time back.</p>
</blockquote>
<p>Did some googling and found out <code class="language-plaintext highlighter-rouge">byzanz-record</code> as the tool perfect for me</p>
<h2 id="installation">Installation</h2>
<p>Beginning for 14.04 and above, it is available in the <code class="language-plaintext highlighter-rouge">universe</code> repository</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>byzanz</code></pre></figure>
<p>If you are on a system older than that</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo </span>add-apt-repository ppa:fossfreedom/byzanz
<span class="nv">$ </span><span class="nb">sudo </span>apt-get update <span class="o">&&</span> <span class="nb">sudo </span>apt-get <span class="nb">install </span>byzanz</code></pre></figure>
<h2 id="usage">Usage:</h2>
<p>We are gonna be using this tool from the command prompt itself as GUI’s would just slow down the process. Now for that we just need four things</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">--x=<your_value></code></li>
<li><code class="language-plaintext highlighter-rouge">--y=<your_value></code></li>
<li><code class="language-plaintext highlighter-rouge">--width=<your_value></code></li>
<li><code class="language-plaintext highlighter-rouge">--height=<your_value></code></li>
</ul>
<p>Now how do we get that?</p>
<p>run</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>xwininfo</code></pre></figure>
<p>and point on the window which you want to record. And it will return you the required values and a little extra information too!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Desktop/pyCalc<span class="nv">$ </span>xwininfo
xwininfo: Please <span class="k">select </span>the window about which you
would like information by clicking the
mouse <span class="k">in </span>that window.
xwininfo: Window <span class="nb">id</span>: 0x740003f <span class="s2">"Calculator"</span>
Absolute upper-left X: 984
Absolute upper-left Y: 509
Relative upper-left X: 0
Relative upper-left Y: 0
Width: 284
Height: 169
Depth: 24
Visual: 0x20
Visual Class: TrueColor
Border width: 0
Class: InputOutput
Colormap: 0x22 <span class="o">(</span>installed<span class="o">)</span>
Bit Gravity State: NorthWestGravity
Window Gravity State: NorthWestGravity
Backing Store State: NotUseful
Save Under State: no
Map State: IsViewable
Override Redirect State: no
Corners: +984+509 <span class="nt">-98</span>+509 <span class="nt">-98-90</span> +984-90
<span class="nt">-geometry</span> 284x169-88-80
tasdik@Acer:~/Desktop/pyCalc<span class="err">$</span></code></pre></figure>
<p>There now, I have got my values</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Desktop/pyCalc<span class="nv">$ </span>byzanz-record <span class="nt">--duration</span><span class="o">=</span>45 <span class="nt">--x</span><span class="o">=</span>984 <span class="nt">--y</span><span class="o">=</span>509 <span class="nt">--width</span><span class="o">=</span>290 <span class="nt">--height</span><span class="o">=</span>170 out.gif</code></pre></figure>
<p>This will immediately start the recording for the <code class="language-plaintext highlighter-rouge">Calculator</code> window until 45 seconds. So be sure to finish whatever you want to do before that.</p>
<p>Here’s is what I got from the above script</p>
<p><img src="https://raw.githubusercontent.com/tasdikrahman/pyCalc/master/pyCalc_usage.gif" alt="https://raw.githubusercontent.com/tasdikrahman/pyCalc/master/pyCalc_usage.gif" /></p>
<h2 id="references">References:</h2>
<ul>
<li><a href="http://askubuntu.com/questions/107726/how-to-create-animated-gif-images-of-a-screencast">http://askubuntu.com/questions/107726/how-to-create-animated-gif-images-of-a-screencast</a></li>
<li><a href="http://askubuntu.com/questions/4428/how-to-create-a-screencast">http://askubuntu.com/questions/4428/how-to-create-a-screencast</a></li>
</ul>
Converting python script into an executable
2015-11-07T00:00:00+00:00
https://www.tasdikrahman.com/2015/11/07/converting-python-script-into-executable
<h2 id="backdrop">BackDrop:</h2>
<p>So recently I was building a Calculator using <code class="language-plaintext highlighter-rouge">tkinter</code>.</p>
<blockquote>
<p>I wrote a <a href="http://www.tasdikrahman.com/2015/11/06/Building-a-calculator/">blog post</a> for the same some time back.</p>
</blockquote>
<p>Now I thought, how awesome it would be if I could distribute it to my friends and let the use it. Problem was that some of them not being CS grads would not know head or tails about how to run it!</p>
<p>So I thought the best way and the easiest way was to convert the <code class="language-plaintext highlighter-rouge">pyCalc.py</code> into a <code class="language-plaintext highlighter-rouge">.exe</code> file.</p>
<p>That way both my purposes were solved.</p>
<ul>
<li>non-cs people would get an interface to run it which was familiar to them. Heck even a granny who knows how to use chrome to watch cookery shows can use it now. Ok, that was a little bit too much</li>
<li>They didn’t have to install anything in this process</li>
</ul>
<h2 id="enter-pyinstaller">Enter <a href="https://github.com/pyinstaller/pyinstaller/">pyinstaller</a></h2>
<blockquote>
<p>Note: Before installing PyInstaller on Windows, you will need to install <code class="language-plaintext highlighter-rouge">PyWin32</code>. You do not need to do this for GNU/Linux or Mac OS X systems.</p>
</blockquote>
<p>To install it. You just have to do</p>
<p><code class="language-plaintext highlighter-rouge">$ sudo pip install pyinstaller</code> for <code class="language-plaintext highlighter-rouge">python2.*</code></p>
<p>or</p>
<p><code class="language-plaintext highlighter-rouge">$ sudo pip3 install pyinstaller</code> for <code class="language-plaintext highlighter-rouge">python3.*</code></p>
<p>If you are behind a proxy server, just add <code class="language-plaintext highlighter-rouge">-E</code> flag like this <code class="language-plaintext highlighter-rouge">sudo -E pip3 ..</code></p>
<h2 id="creating-the-executable">Creating the executable</h2>
<p>I have my <code class="language-plaintext highlighter-rouge">pyCalc.py</code> which I want to make an executable of, in</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Desktop/pyCalc<span class="nv">$ </span>tree
<span class="nb">.</span>
└── pyCalc.py
0 directories, 1 file
tasdik@Acer:~/Desktop/pyCalc<span class="err">$</span></code></pre></figure>
<p>To build the executable</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Desktop/pyCalc<span class="nv">$ </span>pyinstaller <span class="nt">--onefile</span> <span class="nt">--windowed</span> pyCalc.py</code></pre></figure>
<p>Yes, it’s that easy!</p>
<p>If you are not haunted with any errors. You should see two folders being placed in <code class="language-plaintext highlighter-rouge">pyCalc</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Desktop/pyCalc<span class="nv">$ </span>tree
<span class="nb">.</span>
├── build
│ └── pyCalc
│ ├── base_library.zip
│ ├── localpycos
│ │ ├── pyimod01_os_path.pyc
│ │ ├── pyimod02_archive.pyc
│ │ ├── pyimod03_importers.pyc
│ │ └── struct.pyo
│ ├── out00-Analysis.toc
│ ├── out00-EXE.toc
│ ├── out00-PKG.pkg
│ ├── out00-PKG.toc
│ ├── out00-PYZ.pyz
│ ├── out00-PYZ.toc
│ ├── out00-Tree.toc
│ ├── out01-Tree.toc
│ └── warnpyCalc.txt
├── dist
│ └── pyCalc
├── pyCalc.py
└── pyCalc.spec
4 directories, 17 files
tasdik@Acer:~/Desktop/pyCalc<span class="nv">$ </span></code></pre></figure>
<p>For a successful build , the final executable, <code class="language-plaintext highlighter-rouge">pyCalc</code>, and any associated files, will be placed in the <code class="language-plaintext highlighter-rouge">dist</code> directory, which will be created if it doesn’t exist.</p>
<p>Let me briefly describe the options that are being used:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">--onefile</code> is used to package everything into a single executable. If you do not specify this option, the libraries, etc. will be distributed as separate files alongside the main executable.</li>
<li><code class="language-plaintext highlighter-rouge">--windowed</code> prevents a console window from being displayed when the application is run. If you’re releasing a non-graphical application (i.e. a console application), you do not need to use this option.</li>
<li><code class="language-plaintext highlighter-rouge">pyCalc.py</code> the main source file of the application. The basename of this script will be used to name of the executable, however you may specify an alternative executable name using the <code class="language-plaintext highlighter-rouge">--name</code> option.</li>
</ul>
<p>See the PyInstaller Manual for more configuration information.</p>
<p>Sadly, we can’t make <code class="language-plaintext highlighter-rouge">windows</code> executables from <code class="language-plaintext highlighter-rouge">pyinstaller</code>! It supported it some time back in the earlier versions but not in the newer versions.</p>
<h2 id="alternatives-to-pyinstaller">Alternatives to <code class="language-plaintext highlighter-rouge">pyinstaller</code></h2>
<ul>
<li><a href="http://www.py2exe.org/">http://www.py2exe.org/</a></li>
<li><a href="http://nuitka.net/">http://nuitka.net/</a></li>
<li><a href="http://wiki.python.org/moin/Py2Exe">http://wiki.python.org/moin/Py2Exe</a></li>
<li><a href="http://cx-freeze.sourceforge.net/">http://cx-freeze.sourceforge.net/</a></li>
</ul>
<p>Till then. Goodbye!</p>
<h2 id="references">References:</h2>
<ul>
<li><a href="https://mborgerson.com/creating-an-executable-from-a-python-script">https://mborgerson.com/creating-an-executable-from-a-python-script</a></li>
<li><a href="http://stackoverflow.com/questions/5458048/how-to-make-a-python-script-standalone-executable-to-run-without-any-dependency">http://stackoverflow.com/questions/5458048/how-to-make-a-python-script-standalone-executable-to-run-without-any-dependency</a></li>
</ul>
How hard can building a Calculator be right?
2015-11-06T00:00:00+00:00
https://www.tasdikrahman.com/2015/11/06/Building-a-calculator
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" />
<h2 id="backdrop">BackDrop:</h2>
<p>So I was reading about this wonderful module called <code class="language-plaintext highlighter-rouge">tkinter</code> some days back. And man was I not sucked into it!</p>
<p>Sure, we do have some very good GUI modules like <code class="language-plaintext highlighter-rouge">PyQT</code>, <code class="language-plaintext highlighter-rouge">wxPython</code>, <code class="language-plaintext highlighter-rouge">PySide</code>. And I don’t deny the fact that some are way better than <code class="language-plaintext highlighter-rouge">tkinter</code>. But one advantage which makes <code class="language-plaintext highlighter-rouge">tkinter</code> stand out from the crowd is that, it comes pre-packaged with the python package.</p>
<p>What I mean by that is you don’t have to install anything to run and write GUI’s with it and run my Calculator program for instance.</p>
<h2 id="dabbling-with-it">Dabbling with it:</h2>
<p>I got naturally interested in it so after reading the docs for some time. I thought why not apply it by making something small. It got me thinking about what to make with it!</p>
<p>After some rambling aroung, I settled with the thought of making a Calculator using <code class="language-plaintext highlighter-rouge">Tkinter</code>. Aiming for a file search program at my second go.</p>
<h2 id="taking-a-piece-of-paper">Taking a piece of paper:</h2>
<p>So the first task was to decide whether to use <code class="language-plaintext highlighter-rouge">OOP</code>’s for this project or not. I figured out that adding <code class="language-plaintext highlighter-rouge">class</code>‘es would only complicate it in the first go.</p>
<p>Second was to decide which functions/operators should it have initially, upon which initial design should be made.</p>
<p>The problem which took me the hardest to figure out was where we had to parse the input entered by the user.
Take this input as an example.</p>
<p><img src="https://raw.githubusercontent.com/tasdikrahman/www.tasdikrahman.com/master/images/calcBlog_1.jpg" alt="calcBlog_1" /></p>
<p>Now to get <code class="language-plaintext highlighter-rouge">80</code> as the first operand, one has to press <code class="language-plaintext highlighter-rouge">8</code> and then <code class="language-plaintext highlighter-rouge">8</code> again. Now you and I know that it is <code class="language-plaintext highlighter-rouge">80</code>. But how do you make the program understand that?</p>
<p>I figured out that, it was best that I dumped the whole input into a function called <code class="language-plaintext highlighter-rouge">calculate</code> where I got the whole of the entered input through <code class="language-plaintext highlighter-rouge">display.get()</code>, <code class="language-plaintext highlighter-rouge">display</code> being an <code class="language-plaintext highlighter-rouge">Entry()</code> object.</p>
<h3 id="how-was-i-dumping-what-i-clicked-into-entry-widget">How was I dumping what I clicked into <code class="language-plaintext highlighter-rouge">Entry()</code> widget?</h3>
<p>I used two functions for that, <code class="language-plaintext highlighter-rouge">get_variables(num)</code> for <code class="language-plaintext highlighter-rouge">operand</code> and <code class="language-plaintext highlighter-rouge">get_operation(operator)</code> for <code class="language-plaintext highlighter-rouge">operator</code>. Inside each, I have a <code class="language-plaintext highlighter-rouge">global</code> variable, whose value gets incremented each time control transfers to any of these functions. I use this <code class="language-plaintext highlighter-rouge">global</code> variable to keep track of the position of the next data item(be it a operand or operator) to be inserted into the <code class="language-plaintext highlighter-rouge">Entry</code> widget.</p>
<p>Here is what I did for <code class="language-plaintext highlighter-rouge">get_variables()</code></p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">get_variables</span><span class="p">(</span><span class="n">num</span><span class="p">):</span>
<span class="s">"""Gets the user input for operands and puts it inside the entry widget"""</span>
<span class="k">global</span> <span class="n">i</span>
<span class="n">display</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">num</span><span class="p">)</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span></code></pre></figure>
<p>So for a typical <code class="language-plaintext highlighter-rouge">Button</code>, lets take <code class="language-plaintext highlighter-rouge">7</code> here. I have</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">seven</span> <span class="o">=</span> <span class="n">Button</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">text</span> <span class="o">=</span> <span class="s">"7"</span><span class="p">,</span> <span class="n">command</span> <span class="o">=</span> <span class="k">lambda</span> <span class="p">:</span> <span class="n">get_variables</span><span class="p">(</span><span class="mi">7</span><span class="p">),</span> <span class="n">font</span><span class="o">=</span><span class="n">FONT_LARGE</span><span class="p">)</span>
<span class="n">seven</span><span class="p">.</span><span class="n">grid</span><span class="p">(</span><span class="n">row</span> <span class="o">=</span> <span class="mi">4</span><span class="p">,</span> <span class="n">column</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span></code></pre></figure>
<p>Well one problem solved!</p>
<h3 id="adding-and---undo-button">Adding and <code class="language-plaintext highlighter-rouge"><-</code> (undo) button</h3>
<p>Now what if you pressed something wrong, you don’t want to press the <code class="language-plaintext highlighter-rouge">AC</code> button to clear the whole of the entered text as you would have to again waster your time into typing it again. How do we achieve that?</p>
<p>I figured out that the <code class="language-plaintext highlighter-rouge">whole_string</code> stores the whole of the input and what I wanted with the <code class="language-plaintext highlighter-rouge"><-</code> button was an undo of what I did last.</p>
<p>So I just had to remove the last index of the string <code class="language-plaintext highlighter-rouge">whole_string</code>
For that.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">new_string</span> <span class="o">=</span> <span class="n">whole_string</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">print</span><span class="p">(</span><span class="n">new_string</span><span class="p">)</span>
<span class="n">clear_all</span><span class="p">()</span>
<span class="n">display</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">new_string</span><span class="p">)</span></code></pre></figure>
<p>That did it what <code class="language-plaintext highlighter-rouge"><-</code> was supposed to do</p>
<h3 id="how-do-i-seperate-the-operands-from-the-operators">How do I seperate the operands from the operators?</h3>
<p>Now at first thoughts, I thought I should hardcode it for each and every operator. Like you have <code class="language-plaintext highlighter-rouge">+</code> inside the <code class="language-plaintext highlighter-rouge">whole_string</code> which stores the value of <code class="language-plaintext highlighter-rouge">display.get()</code>. And then you do a</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">if</span> <span class="s">'+'</span> <span class="ow">in</span> <span class="n">whole_string</span><span class="p">:</span>
<span class="c1"># Now to split the contents into `operands`, I did a `var1, var2 = whole_string.split(operator)`.
</span> <span class="n">var1</span><span class="p">,</span> <span class="n">var2</span> <span class="o">=</span> <span class="n">whole_string</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"+"</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">var1</span><span class="p">)</span> <span class="o">+</span> <span class="nb">int</span><span class="p">(</span><span class="n">var2</span><span class="p">)</span></code></pre></figure>
<p>But you notice that this fails at the slightest of sneezes. Enter more than 2 variables and then you are left with an extra unused operand. You will left making <code class="language-plaintext highlighter-rouge">if-else</code> clauses for the rest of this project.</p>
<p>So I dropped this approach for good!</p>
<p>After some frustrating 3 hours and messing around and deleting with 3 <code class="language-plaintext highlighter-rouge">branches</code>. I finally came to this.</p>
<blockquote>
<p>Why not user <code class="language-plaintext highlighter-rouge">parser</code> to parse the expression and evaluate it.</p>
</blockquote>
<p>So this is what I did</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">calculate</span><span class="p">():</span>
<span class="n">whole_string</span> <span class="o">=</span> <span class="n">display</span><span class="p">.</span><span class="n">get</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">formulae</span> <span class="o">=</span> <span class="n">parser</span><span class="p">.</span><span class="n">expr</span><span class="p">(</span><span class="n">whole_string</span><span class="p">).</span><span class="nb">compile</span><span class="p">()</span> <span class="c1">## returns a `parser` object
</span> <span class="n">result</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="n">formulae</span><span class="p">)</span> <span class="c1">## evaluates the parsed expression
</span> <span class="n">clear_all</span><span class="p">()</span>
<span class="n">display</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">Exception</span><span class="p">:</span>
<span class="n">clear_all</span><span class="p">()</span>
<span class="n">display</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="s">"Error!"</span><span class="p">)</span></code></pre></figure>
<p>This solved my problem and it works for most of the test cases I have checked with, but by far the best approach would be write my own <code class="language-plaintext highlighter-rouge">parser</code>.</p>
<p>Will refactor to include it in my next release.</p>
<h2 id="download-it">Download it!</h2>
<p><a href="https://github.com/tasdikrahman/pyCalc/releases/download/v1.0/pyCalc_v1" target="_blank"><i class="fa fa-linux fa-2x"></i></a></p>
<h2 id="fork-this-project">Fork this project</h2>
<p>Feel free to fork this project and make changes to it.</p>
<div><a href="https://github.com/tasdikrahman/pyCalc" class="btn btn-info">Github repo</a></div>
<h2 id="filing-bugs">Filing bugs</h2>
<p>If you find some bugs,
please file an issue on the github page</p>
<div><a href="https://github.com/tasdikrahman/pyCalc/issues/new" class="btn btn-danger">Report a bug</a></div>
<p>Till then. Goodbye!</p>
Setting up DBPedia Spotlight on your local server
2015-10-24T00:00:00+00:00
https://www.tasdikrahman.com/2015/10/24/Setting-up-DBPedia-on-your-local-server
<h2 id="intro-">Intro :</h2>
<p>DBPedia Spotlight can be queried over the API which they provide. But for our convenience, it is not always possible to do so.</p>
<p>So setting it up locally was the best solution.</p>
<p>You can run DBpedia Spotlight from the comfort of your own machine in one or many of the following ways:</p>
<ul>
<li><strong>Web Service</strong> (no installation needed). We offer a WADL service descriptor, so with Eclipse or Netbeans you can automagically create a client to call our Web Service. See: Web Service</li>
<li><strong>JAR</strong>. We offer a jar with all dependencies included. You can just download it and run from command line. See: Run from a Jar</li>
<li><strong>Maven</strong>. Our build is mavenized, which means that you can use the scala plugin to run our classes from command line. See: Build from Source with Maven</li>
<li><strong>Ubuntu/Debian package</strong>. We are also starting to share our downloads as debian packages so that anybody can just install DBpedia Spotlight directly from apt-get, Synaptic or their favorite package manager. See:Debian-Package-Installation:-How-To</li>
<li><strong>WAR Files/ Tomcat</strong>. DBpedia Spotlight is also build as a WAR file. You can use it through Apache Tomcat.</li>
</ul>
<p>We would be doing it the <strong>JAR</strong> way.</p>
<h2 id="get-set-go">Get set. Go</h2>
<p>Requirements</p>
<ul>
<li>Java 1.6+</li>
<li>RAM of appropriate size for the spotter lexicon you need</li>
</ul>
<p>First we will install a pre-packaged lightweight deployment to get you started.</p>
<p><strong>Lucene</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sys2@sys2:~<span class="nv">$ </span>wget http://spotlight.dbpedia.org/download/release-0.6/dbpedia-spotlight-quickstart-0.6.5.zip
<span class="nt">--2015-10-22</span> 10:28:16-- http://spotlight.dbpedia.org/download/release-0.6/dbpedia-spotlight-quickstart-0.6.5.zip
Resolving spotlight.dbpedia.org <span class="o">(</span>spotlight.dbpedia.org<span class="o">)</span>... 134.155.95.15
Connecting to spotlight.dbpedia.org <span class="o">(</span>spotlight.dbpedia.org<span class="o">)</span>|134.155.95.15|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://wifo5-04.informatik.uni-mannheim.de/downloads/release-0.6/dbpedia-spotlight-quickstart-0.6.5.zip <span class="o">[</span>following]
<span class="nt">--2015-10-22</span> 10:28:17-- http://wifo5-04.informatik.uni-mannheim.de/downloads/release-0.6/dbpedia-spotlight-quickstart-0.6.5.zip
Resolving wifo5-04.informatik.uni-mannheim.de <span class="o">(</span>wifo5-04.informatik.uni-mannheim.de<span class="o">)</span>... 134.155.95.17
Connecting to wifo5-04.informatik.uni-mannheim.de <span class="o">(</span>wifo5-04.informatik.uni-mannheim.de<span class="o">)</span>|134.155.95.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 156569414 <span class="o">(</span>149M<span class="o">)</span> <span class="o">[</span>application/zip]
Saving to: ‘dbpedia-spotlight-quickstart-0.6.5.zip’
100%[<span class="o">====================================================================================================>]</span> 15,65,69,414 1.14MB/s <span class="k">in </span>5m 51s
2015-10-22 10:34:09 <span class="o">(</span>435 KB/s<span class="o">)</span> - ‘dbpedia-spotlight-quickstart-0.6.5.zip’ saved <span class="o">[</span>156569414/156569414]
sys2@sys2:~<span class="err">$</span>
sys2@sys2:~<span class="nv">$ </span>unzip dbpedia-spotlight-quickstart-0.6.5.zip
Archive: dbpedia-spotlight-quickstart-0.6.5.zip
creating: dbpedia-spotlight-quickstart-0.6.5/data/
creating: dbpedia-spotlight-quickstart-0.6.5/data/index/
inflating: dbpedia-spotlight-quickstart-0.6.5/data/index/_1.cfs
inflating: dbpedia-spotlight-quickstart-0.6.5/data/index/_2.cfs
inflating: dbpedia-spotlight-quickstart-0.6.5/data/index/_3.cfs
inflating: dbpedia-spotlight-quickstart-0.6.5/data/index/segments.gen
inflating: dbpedia-spotlight-quickstart-0.6.5/data/index/segments_5
inflating: dbpedia-spotlight-quickstart-0.6.5/data/index/similarity-thresholds.txt
creating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/
creating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-chunker.bin
extracting: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-chunker.zip
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-ner-location.bin
extracting: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-ner-location.zip
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-ner-organization.bin
extracting: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-ner-organization.zip
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-ner-person.bin
extracting: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-ner-person.zip
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-pos-maxent.bin
extracting: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-pos-maxent.zip
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-sent.bin
extracting: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-sent.zip
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-token.bin
extracting: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/english/en-token.zip
inflating: dbpedia-spotlight-quickstart-0.6.5/data/opennlp/README.txt
inflating: dbpedia-spotlight-quickstart-0.6.5/data/pos-en-general-brown.HiddenMarkovModel
inflating: dbpedia-spotlight-quickstart-0.6.5/data/spotter.dict
inflating: dbpedia-spotlight-quickstart-0.6.5/dbpedia-spotlight-0.6.5-jar-with-dependencies.jar
inflating: dbpedia-spotlight-quickstart-0.6.5/run.sh
inflating: dbpedia-spotlight-quickstart-0.6.5/server.properties
inflating: dbpedia-spotlight-quickstart-0.6.5/apache-2.0.txt
inflating: dbpedia-spotlight-quickstart-0.6.5/lingpipe-license-1.txt
sys2@sys2:~<span class="nv">$ </span><span class="nb">cd </span>dbpedia-spotlight-quickstart-0.6.5/
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5<span class="nv">$ </span></code></pre></figure>
<p><strong>Statistical</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sys2@sys2:~<span class="nv">$ </span>wget http://spotlight.sztaki.hu/downloads/version-0.1/en.tar.gz
<span class="nt">--2015-10-22</span> 11:12:58-- http://spotlight.sztaki.hu/downloads/version-0.1/en.tar.gz
Resolving spotlight.sztaki.hu <span class="o">(</span>spotlight.sztaki.hu<span class="o">)</span>... 193.225.89.3
Connecting to spotlight.sztaki.hu <span class="o">(</span>spotlight.sztaki.hu<span class="o">)</span>|193.225.89.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2257455959 <span class="o">(</span>2.1G<span class="o">)</span> <span class="o">[</span>application/x-gzip]
Saving to: ‘en.tar.gz’
100%[<span class="o">==================================================================================================>]</span> 2,25,74,55,959 935KB/s <span class="k">in </span>44m 59s
2015-10-22 11:57:58 <span class="o">(</span>817 KB/s<span class="o">)</span> - ‘en.tar.gz’ saved <span class="o">[</span>2257455959/2257455959]
sys2@sys2:~<span class="nv">$ </span>
sys2@sys2:~<span class="nv">$ </span>wget http://spotlight.sztaki.hu/downloads/version-0.1/dbpedia-spotlight.jar
<span class="nt">--2015-10-22</span> 12:13:39-- http://spotlight.sztaki.hu/downloads/version-0.1/dbpedia-spotlight.jar
Resolving spotlight.sztaki.hu <span class="o">(</span>spotlight.sztaki.hu<span class="o">)</span>... 193.225.89.3
Connecting to spotlight.sztaki.hu <span class="o">(</span>spotlight.sztaki.hu<span class="o">)</span>|193.225.89.3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 116325524 <span class="o">(</span>111M<span class="o">)</span> <span class="o">[</span>application/java-archive]
Saving to: ‘dbpedia-spotlight.jar’
100%[<span class="o">====================================================================================================>]</span> 11,63,25,524 415KB/s <span class="k">in </span>3m 27s
2015-10-22 12:17:07 <span class="o">(</span>549 KB/s<span class="o">)</span> - ‘dbpedia-spotlight.jar’ saved <span class="o">[</span>116325524/116325524]
sys2@sys2:~<span class="nv">$ </span><span class="nb">tar </span>xvf en.tar.gz
en/
en/model/
en/model/res.mem
en/model/res.mem_
en/model/tokens.mem
en/model/context.mem
en/model/sf.mem
en/model/candmap.mem
en/model.properties
en/stopwords.list
en/spotter_thresholds.txt
en/opennlp/
en/opennlp/pos-maxent.bin
en/opennlp/token.bin
en/opennlp/chunker.bin
en/opennlp/sent.bin
sys2@sys2:~<span class="err">$</span>
sys2@sys2:~<span class="err">$</span></code></pre></figure>
<h2 id="add-the-data-corpus">Add the Data corpus</h2>
<p>We can run the model right now, but I will do so after adding the larger data corpus.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5<span class="nv">$ </span><span class="nb">cd </span>data
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span>wget http://spotlight.dbpedia.org/download/release-0.5/context-index-compact.tgz
<span class="nt">--2015-10-22</span> 12:25:58-- http://spotlight.dbpedia.org/download/release-0.5/context-index-compact.tgz
Resolving spotlight.dbpedia.org <span class="o">(</span>spotlight.dbpedia.org<span class="o">)</span>... 134.155.95.15
Connecting to spotlight.dbpedia.org <span class="o">(</span>spotlight.dbpedia.org<span class="o">)</span>|134.155.95.15|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://wifo5-04.informatik.uni-mannheim.de/downloads/release-0.5/context-index-compact.tgz <span class="o">[</span>following]
<span class="nt">--2015-10-22</span> 12:25:59-- http://wifo5-04.informatik.uni-mannheim.de/downloads/release-0.5/context-index-compact.tgz
Resolving wifo5-04.informatik.uni-mannheim.de <span class="o">(</span>wifo5-04.informatik.uni-mannheim.de<span class="o">)</span>... 134.155.95.17
Connecting to wifo5-04.informatik.uni-mannheim.de <span class="o">(</span>wifo5-04.informatik.uni-mannheim.de<span class="o">)</span>|134.155.95.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12017976481 <span class="o">(</span>11G<span class="o">)</span> <span class="o">[</span>application/x-gzip]
Saving to: ‘context-index-compact.tgz’
100%[<span class="o">=================================================================================================>]</span> 12,01,79,76,481 1.16MB/s <span class="k">in </span>4h 10m
2015-10-22 16:36:37 <span class="o">(</span>781 KB/s<span class="o">)</span> - ‘context-index-compact.tgz’ saved <span class="o">[</span>12017976481/12017976481]
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span><span class="nb">tar </span>zxvf context-index-compact.tgz
index-withSF-withTypes-compressed/
index-withSF-withTypes-compressed/_at.cfs
index-withSF-withTypes-compressed/_66.cfs
index-withSF-withTypes-compressed/segments_9t
index-withSF-withTypes-compressed/_99.cfs
index-withSF-withTypes-compressed/similarity-thresholds.txt
index-withSF-withTypes-compressed/segments.gen
index-withSF-withTypes-compressed/_33.cfs
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span><span class="nb">mv </span>index-withSF-withTypes-compressed index
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span>wget http://spotlight.dbpedia.org/download/release-0.4/surface_forms-Wikipedia-TitRedDis.uriThresh75.tsv.spotterDictionary.gz
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span><span class="nb">gunzip </span>surface_forms-Wikipedia-TitRedDis.uriThresh75.tsv.spotterDictionary.gz
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span><span class="nb">mv </span>surface_forms-Wikipedia-TitRedDis.uriThresh75.tsv.spotterDictionary spotter.dict
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span></code></pre></figure>
<h2 id="testing-the-installation-">Testing the installation :</h2>
<p>In order to test the installation, do a</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span>curl http://sys2:2222/rest/annotate <span class="nt">-H</span> <span class="s2">"Accept: text/xml"</span> <span class="nt">--data-urlencode</span> <span class="s2">"text=The earliest authenticated human remains in South Asia date to about 30,000 years ago.[26] Nearly contemporaneous Mesolithic rock art sites have been found in many parts of the Indian subcontinent, including at the Bhimbetka rock shelters in Madhya Pradesh.[27] Around 7000 BCE, the first known Neolithic settlements appeared on the subcontinent in Mehrgarh and other sites in western Pakistan.[28] These gradually developed into the Indus Valley Civilisation,[29] the first urban culture in South Asia;[30] it flourished during 2500–1900 BCE in Pakistan and western India along the river valleys of Indus and Sarasvati.[31] Centred on cities such as Mohenjo-daro, Harappa, Rakhigarhi, Dholavira, and Kalibangan, and relying on varied forms of subsistence, the civilisation engaged robustly in crafts production and wide-ranging trade."</span> <span class="nt">--data</span> <span class="s2">"confidence=0"</span> <span class="nt">--data</span> <span class="s2">"support=0"</span>
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="nv">$ </span>curl http://sys2:2222/rest/annotate <span class="nt">-H</span> <span class="s2">"Accept: text/xml"</span> <span class="nt">--data-urlencode</span> <span class="s2">"text=The earliest authenticated human remains in South Asia date to about 30,000 years ago.[26] Nearly contemporaneous Mesolithic rock art sites have been found in many parts of the Indian subcontinent, including at the Bhimbetka rock shelters in Madhya Pradesh.[27] Around 7000 BCE, the first known Neolithic settlements appeared on the subcontinent in Mehrgarh and other sites in western Pakistan.[28] These gradually developed into the Indus Valley Civilisation,[29] the first urban culture in South Asia;[30] it flourished during 2500–1900 BCE in Pakistan and western India along the river valleys of Indus and Sarasvati.[31] Centred on cities such as Mohenjo-daro, Harappa, Rakhigarhi, Dholavira, and Kalibangan, and relying on varied forms of subsistence, the civilisation engaged robustly in crafts production and wide-ranging trade."</span> <span class="nt">--data</span> <span class="s2">"confidence=0"</span> <span class="nt">--data</span> <span class="s2">"support=0"</span>
<?xml <span class="nv">version</span><span class="o">=</span><span class="s2">"1.0"</span> <span class="nv">encoding</span><span class="o">=</span><span class="s2">"utf-8"</span>?>
<Annotation <span class="nv">text</span><span class="o">=</span><span class="s2">"The earliest authenticated human remains in South Asia date to about 30,000 years ago.[26] Nearly contemporaneous Mesolithic rock art sites have been found in many parts of the Indian subcontinent, including at the Bhimbetka rock shelters in Madhya Pradesh.[27] Around 7000 BCE, the first known Neolithic settlements appeared on the subcontinent in Mehrgarh and other sites in western Pakistan.[28] These gradually developed into the Indus Valley Civilisation,[29] the first urban culture in South Asia;[30] it flourished during 2500–1900 BCE in Pakistan and western India along the river valleys of Indus and Sarasvati.[31] Centred on cities such as Mohenjo-daro, Harappa, Rakhigarhi, Dholavira, and Kalibangan, and relying on varied forms of subsistence, the civilisation engaged robustly in crafts production and wide-ranging trade."</span> <span class="nv">confidence</span><span class="o">=</span><span class="s2">"0.0"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"0"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">""</span> <span class="nv">sparql</span><span class="o">=</span><span class="s2">""</span> <span class="nv">policy</span><span class="o">=</span><span class="s2">"whitelist"</span><span class="o">></span>
<Resources>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/South_Asia"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"2850"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"Freebase:/book/book_subject,Freebase:/book,Freebase:/location/location,Freebase:/location,Freebase:/organization/organization_scope,Freebase:/organization,Freebase:/location/region,Freebase:/people/ethnicity,Freebase:/people"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"South Asia"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"44"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.13776831328868866"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Mesolithic"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"681"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">""</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Mesolithic"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"114"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.13642436265945435"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Indian_subcontinent"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"1497"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"DBpedia:Continent,DBpedia:PopulatedPlace,DBpedia:Place,Schema:Place,Schema:Continent,Freebase:/location/region,Freebase:/location,Freebase:/location/location"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Indian subcontinent"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"177"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.141897514462471"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Madhya_Pradesh"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"2950"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"DBpedia:Settlement,DBpedia:PopulatedPlace,DBpedia:Place,Schema:Place,Freebase:/location/in_state,Freebase:/location,Freebase:/location/statistical_region,Freebase:/location/dated_location,Freebase:/location/location,Freebase:/book/author,Freebase:/book,Freebase:/location/administrative_division"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Madhya Pradesh"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"242"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.13563162088394165"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Common_Era"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"1247"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">""</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"BCE"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"274"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.1675749570131302"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"0.20302364934710979"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Neolithic"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"2903"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">""</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Neolithic"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"295"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.11610618233680725"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Pakistan"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"23561"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"DBpedia:Country,DBpedia:PopulatedPlace,DBpedia:Place,Schema:Place,Schema:Country,Freebase:/location/country,Freebase:/location,Freebase:/organization/organization_member,Freebase:/organization,Freebase:/biology/breed_origin,Freebase:/biology,Freebase:/location/statistical_region,Freebase:/military/military_combatant,Freebase:/military,Freebase:/people/ethnicity,Freebase:/people,Freebase:/location/dated_location,Freebase:/law/court_jurisdiction_area,Freebase:/law,Freebase:/sports/sport_country,Freebase:/sports,Freebase:/government/governmental_jurisdiction,Freebase:/government,Freebase:/olympics/olympic_participating_country,Freebase:/olympics,Freebase:/location/location,Freebase:/meteorology/cyclone_affected_area,Freebase:/meteorology,Freebase:/book/book_subject,Freebase:/book,Freebase:/sports/sports_team_location"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Pakistan"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"385"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.0921529158949852"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"0.6893670279187828"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Indus_Valley_Civilization"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"424"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"Freebase:/time/event,Freebase:/time,Freebase:/book/book_subject,Freebase:/book"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Indus Valley Civilisation"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"434"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.20086094737052917"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/South_Asia"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"2850"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"Freebase:/book/book_subject,Freebase:/book,Freebase:/location/location,Freebase:/location,Freebase:/organization/organization_scope,Freebase:/organization,Freebase:/location/region,Freebase:/people/ethnicity,Freebase:/people"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"South Asia"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"492"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.13776831328868866"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Common_Era"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"1247"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">""</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"BCE"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"539"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.1675749570131302"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"0.20302364934710979"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Pakistan"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"23561"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"DBpedia:Country,DBpedia:PopulatedPlace,DBpedia:Place,Schema:Place,Schema:Country,Freebase:/location/country,Freebase:/location,Freebase:/organization/organization_member,Freebase:/organization,Freebase:/biology/breed_origin,Freebase:/biology,Freebase:/location/statistical_region,Freebase:/military/military_combatant,Freebase:/military,Freebase:/people/ethnicity,Freebase:/people,Freebase:/location/dated_location,Freebase:/law/court_jurisdiction_area,Freebase:/law,Freebase:/sports/sport_country,Freebase:/sports,Freebase:/government/governmental_jurisdiction,Freebase:/government,Freebase:/olympics/olympic_participating_country,Freebase:/olympics,Freebase:/location/location,Freebase:/meteorology/cyclone_affected_area,Freebase:/meteorology,Freebase:/book/book_subject,Freebase:/book,Freebase:/sports/sports_team_location"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Pakistan"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"546"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.0921529158949852"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"0.6893670279187828"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/South_Asia"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"2850"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"Freebase:/book/book_subject,Freebase:/book,Freebase:/location/location,Freebase:/location,Freebase:/organization/organization_scope,Freebase:/organization,Freebase:/location/region,Freebase:/people/ethnicity,Freebase:/people"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"India"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"567"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.1619909405708313"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"0.8368753166673841"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Indus_Valley_Civilization"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"424"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"Freebase:/time/event,Freebase:/time,Freebase:/book/book_subject,Freebase:/book"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Indus"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"600"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.20086094737052917"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Sarasvati_River"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"60"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"Freebase:/geography/river,Freebase:/geography,Freebase:/location/location,Freebase:/location,Freebase:/geography/geographical_feature,Freebase:/geography/body_of_water,Freebase:/religion/deity,Freebase:/religion"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Sarasvati"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"610"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.08038105070590973"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"0.9255801653640217"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Mohenjo-daro"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"136"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"DBpedia:WorldHeritageSite,DBpedia:Place,Schema:Place,Freebase:/protected_sites/listed_site,Freebase:/protected_sites,Freebase:/location/location,Freebase:/location"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Mohenjo-daro"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"651"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.19224941730499268"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
<Resource <span class="nv">URI</span><span class="o">=</span><span class="s2">"http://dbpedia.org/resource/Harappa"</span> <span class="nv">support</span><span class="o">=</span><span class="s2">"163"</span> <span class="nv">types</span><span class="o">=</span><span class="s2">"DBpedia:Settlement,DBpedia:PopulatedPlace,DBpedia:Place,Schema:Place,Freebase:/location/location,Freebase:/location,Freebase:/location/statistical_region,Freebase:/location/citytown,Freebase:/location/dated_location"</span> <span class="nv">surfaceForm</span><span class="o">=</span><span class="s2">"Harappa"</span> <span class="nv">offset</span><span class="o">=</span><span class="s2">"665"</span> <span class="nv">similarityScore</span><span class="o">=</span><span class="s2">"0.17811940610408783"</span> <span class="nv">percentageOfSecondRank</span><span class="o">=</span><span class="s2">"-1.0"</span>/>
</Resources>
</Annotation>
sys2@sys2:~/dbpedia-spotlight-quickstart-0.6.5/data<span class="err">$</span></code></pre></figure>
<p>So there you go.</p>
<h2 id="references">References</h2>
<ul>
<li>
<p><a href="https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki/Run-from-a-JAR">https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki/Run-from-a-JAR</a></p>
</li>
<li>
<p><a href="https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki/Installation">https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki/Installation</a></p>
</li>
<li>
<p><a href="https://dbpedia-spotlight.github.io/demo/">https://dbpedia-spotlight.github.io/demo/</a></p>
</li>
<li>
<p><a href="https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki">https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki</a></p>
</li>
</ul>
<p>Till then. Goodbye!</p>
Running CGI Scripts with CGIHTTPServer
2015-10-20T00:00:00+00:00
https://www.tasdikrahman.com/2015/10/20/Running-CGI-Scripts-with-CGIHTTPServer
<h2 id="intro-">Intro :</h2>
<p>So why should we be interested in CGI which stands for common gateway interface. Well for instance, try to recall the websites that you have been in the last 1 hour. Now out of those websites, some where static and some were dynamic. The latter would mean that the contents of that website kept changing in real time.</p>
<p>CGI scripting is helpful when you want to generate content from the data residing in a database. This is not only convenient but also cuts a lot of time.</p>
<p>In my previous article, I showed how to run CGI scripts in an <code class="language-plaintext highlighter-rouge">apache2</code> webserver. Here is the link, if you wanna take a look at it</p>
<blockquote>
<p><strong><a href="http://www.tasdikrahman.com/2015/09/30/Running-CGI-Sripts-on-Apache2-Ubuntu/">Running-CGI-Sripts-on-Apache2-Ubuntu</a></strong></p>
</blockquote>
<h2 id="cgihttpserver">CGIHTTPServer</h2>
<p>I hope that you have python installed in your system. Just to be sure.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span>python <span class="nt">--version</span>
Python 2.7.8
tasdik@Acer:~<span class="err">$</span></code></pre></figure>
<p>We will be making use of a super simple Web server shipped by default with <code class="language-plaintext highlighter-rouge">python</code> instead of using a full blown web server software for the sake of understanding.</p>
<p>Now cgi scripts are executable files inside the <code class="language-plaintext highlighter-rouge">cgi-bin</code> or <code class="language-plaintext highlighter-rouge">htdocs</code> directory which the web server executes. After which the output of the program is captured in the standard output to be displayed back by the server.</p>
<p>It is the <code class="language-plaintext highlighter-rouge">cgi-bin</code> directory first where all our executable scripts will reside.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd </span>cgi_demo/
tasdik@Acer:~/cgi_demo<span class="nv">$ </span>tree
<span class="nb">.</span>
├── cgi-bin
│ └── retrieval.py
└── forms.html
1 directory, 2 files
tasdik@Acer:~/cgi_demo<span class="nv">$ </span><span class="nb">chmod</span> +x cgi-bin/retrieval.py</code></pre></figure>
<p>**NOTE: ** <code class="language-plaintext highlighter-rouge">Don't forget to make your cgi-script Executable</code></p>
<p><code class="language-plaintext highlighter-rouge">/forms.html</code></p>
<figure class="highlight"><pre><code class="language-html" data-lang="html"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
</pre></td><td class="code"><pre><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><title></span>A simple form demonstration<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">style=</span><span class="s">"text-align:center;"</span><span class="nt">></span>
<span class="nt"><h1></span>User login<span class="nt"></h1></span>
<span class="nt"><form</span> <span class="na">action=</span><span class="s">"/cgi-bin/retrieval.py"</span> <span class="na">method=</span><span class="s">"get"</span><span class="nt">></span>
username : <span class="nt"><input</span> <span class="na">type=</span><span class="s">"text"</span> <span class="na">name=</span><span class="s">"username"</span> <span class="na">style=</span><span class="s">"text-align:center;"</span><span class="nt">></span>
<span class="nt"><br><br></span>
password : <span class="nt"><input</span> <span class="na">type=</span><span class="s">"password"</span> <span class="na">name=</span><span class="s">"password"</span> <span class="na">style=</span><span class="s">"text-align:center;"</span><span class="nt">></span>
<span class="nt"><br><br><br></span>
<span class="nt"><input</span> <span class="na">type=</span><span class="s">"submit"</span> <span class="na">value=</span><span class="s">"Submit"</span><span class="nt">></span>
<span class="nt"></form></span>
<span class="nt"></div></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</pre></td></tr></tbody></table></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">/cgi-bin/retrieval.py</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
</pre></td><td class="code"><pre><span class="c">#!/usr/bin/env python3.4</span>
import cgi, cgitb
cgitb.enable<span class="o">()</span> <span class="c">## allows for debugging errors from the cgi scripts in the browser</span>
form <span class="o">=</span> cgi.FieldStorage<span class="o">()</span>
<span class="c">## getting the data from the fields </span>
first <span class="o">=</span> form.getvalue<span class="o">(</span><span class="s1">'username'</span><span class="o">)</span>
last <span class="o">=</span> form.getvalue<span class="o">(</span><span class="s1">'password'</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"Content-type:text/html</span><span class="se">\r\n\r\n</span><span class="s2">"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"<html>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"<head><title>User entered</title></head>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"<body>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"<h1>User has entered</h1>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"<b>Firstname : </b>"</span> + first + <span class="s2">"<br>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"<br><b>Lastname : </b>"</span> + last + <span class="s2">"<br>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">""</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"</div>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"</body>"</span><span class="o">)</span>
print<span class="o">(</span><span class="s2">"</html>"</span><span class="o">)</span>
</pre></td></tr></tbody></table></code></pre></figure>
<h2 id="running-the-webserver">Running the webserver</h2>
<p>We will be running our webserver in the <code class="language-plaintext highlighter-rouge">cgi_demo</code> directory.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/cgi_demo<span class="nv">$ </span>python <span class="nt">-m</span> CGIHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...</code></pre></figure>
<p>At this point. Our web server is up and running. To check on it.</p>
<p>Go to the link <strong><code class="language-plaintext highlighter-rouge">http://localhost:8000/forms.html</code></strong> in your browser.</p>
<p>You will be displayed with the forms to enter <code class="language-plaintext highlighter-rouge">name</code> and <code class="language-plaintext highlighter-rouge">password</code>. At this point, if you have a look at the terminal. You will see</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/cgi_demo<span class="nv">$ </span>python <span class="nt">-m</span> CGIHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
127.0.0.1 - - <span class="o">[</span>21/Oct/2015 14:26:32] <span class="s2">"GET /forms.html HTTP/1.1"</span> 200 -</code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">200</code> is the response code for the request made by the browser for the file <code class="language-plaintext highlighter-rouge">forms.html</code>, which was present. So it was served back by the server to the browser.</p>
<p>After filling up the form and submitting it. The form calls <code class="language-plaintext highlighter-rouge">retrieval.py</code> program and passes the data entered by the user to it using a <code class="language-plaintext highlighter-rouge">GET</code> request.</p>
<p>You will be redirected to page with a url looking something like</p>
<p><strong><code class="language-plaintext highlighter-rouge">http://localhost:8000/cgi-bin/retrieval.py?username=tasdik&password=admin123</code></strong></p>
<p>You will notice that the form data is appended with the url of the program itself. This is the standard way of how the <code class="language-plaintext highlighter-rouge">GET</code> method passes data onto functions.</p>
<p>The data part starts after the <code class="language-plaintext highlighter-rouge">?</code></p>
<p>After you are redirected. You will notice a change in your terminal window from where you had started the web server</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/cgi_demo<span class="nv">$ </span>python <span class="nt">-m</span> CGIHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
127.0.0.1 - - <span class="o">[</span>21/Oct/2015 17:17:32] <span class="s2">"GET /forms.html HTTP/1.1"</span> 200 -
127.0.0.1 - - <span class="o">[</span>21/Oct/2015 17:17:37] <span class="s2">"GET /cgi-bin/retrieval.py?username=tasdik&password=admin123 HTTP/1.1"</span> 200 -</code></pre></figure>
<p>The program <code class="language-plaintext highlighter-rouge">retrieval.py</code> was served successfully to the browser.</p>
<p>That’s all for this article. In the next one, I will write about how to talk to an <code class="language-plaintext highlighter-rouge">sqlite3</code> database using CGI scripts</p>
<p>Till then. Goodbye!</p>
Running CGI Scripts on Apache2
2015-09-30T00:00:00+00:00
https://www.tasdikrahman.com/2015/09/30/Running-CGI-Scripts-on-Apache2-Ubuntu
<h2 id="intro">Intro</h2>
<p>Have you ever wanted to create a webpage or process user input from a web-based form using Python? These tasks can be accomplished through the use of Python CGI (Common Gateway Interface) scripts with an Apache web server. CGI scripts are called by a web server when a user requests a particular URL or interacts with the webpage (such as clicking a “Submit” button). After the CGI script is called and finishes executing, the output is used by the web server to create a webpage displayed to the user.</p>
<p>If you just want to test the waters in the CGI world, you might wanna test your scripts in a simple server like the one shipped with python default.</p>
<p>I have written an article on that. Here’s the link.</p>
<p><strong><a href="http://www.tasdikrahman.com/2015/10/20/Running-CGI-Sripts-with-CGIHTTPServer/">Running-CGI-Sripts-with-CGIHTTPServer</a></strong></p>
<h2 id="configuring-the-apache2-web-server-to-run-cgi-scripts">Configuring the Apache2 Web server to run CGI scripts</h2>
<p>I am assuming that you are using <code class="language-plaintext highlighter-rouge">apache2</code> version <code class="language-plaintext highlighter-rouge">2.4.*</code>, as in <code class="language-plaintext highlighter-rouge">Apache2.4</code>, the configuration was cleaned up considerably, and things in the default site definition have been moved to configuration files in conf-available. Among other things, this also includes the CGI-related configuration lines seen in the default site of older versions. These have been moved to <code class="language-plaintext highlighter-rouge">/etc/apache2/conf-available/serve-cgi-bin.conf</code>, which contains:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
</code></pre></div></div>
<p>Just to check whether you have <code class="language-plaintext highlighter-rouge">apache2</code> installed on your system and its up and running do a</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasdik@Acer:~<span class="nv">$ </span>apache2 <span class="nt">-v</span>
Server version: Apache/2.4.10 <span class="o">(</span>Ubuntu<span class="o">)</span>
Server built: Mar 5 2015 18:13:03
tasdik@Acer:~<span class="err">$</span>
</code></pre></div></div>
<p>I am currently on version <code class="language-plaintext highlighter-rouge">2.4</code>. If you don’t get any output from the prompt its likely that you don’t have it installed on your system. Just do a</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>apache2
</code></pre></div></div>
<p>Anyways, You just need to make changes on two configuration files, them being</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/etc/apache2/apache2.conf</code></li>
<li><code class="language-plaintext highlighter-rouge">/etc/apache2/conf-available/serve-cgi-bin.conf</code></li>
</ul>
<p>On the terminal</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasdik@Acer:~<span class="nv">$ </span><span class="nb">mkdir</span> /var/www/cgi-bin
tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd</span> /var/www/cgi-bin/
tasdik@Acer:/var/www/cgi-bin<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/apache2/apache2.conf
</code></pre></div></div>
<p>And add the following at the end</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">###################################################################</span>
<span class="c">######### Adding capaility to run CGI-scripts #################</span>
ServerName localhost
ScriptAlias /cgi-bin/ /var/www/cgi-bin/
Options +ExecCGI
AddHandler cgi-script .cgi .pl .py
</code></pre></div></div>
<p>This converts the <code class="language-plaintext highlighter-rouge">cgi-bin</code> address to the specified address and the <code class="language-plaintext highlighter-rouge">AddHandler</code> tells the server to treat all files with <code class="language-plaintext highlighter-rouge">.cgi</code>,<code class="language-plaintext highlighter-rouge">.pl</code> and <code class="language-plaintext highlighter-rouge">.py</code> extensions as cgi scripts.</p>
<p>Now for the second conf file</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/apache2/conf-available/serve-cgi-bin.conf
</code></pre></div></div>
<p>The final file should look something like this</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><IfModule mod_alias.c>
<IfModule mod_cgi.c>
Define ENABLE_USR_LIB_CGI_BIN
</IfModule>
<IfModule mod_cgid.c>
Define ENABLE_USR_LIB_CGI_BIN
</IfModule>
<IfDefine ENABLE_USR_LIB_CGI_BIN>
<span class="c">#ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/</span>
<span class="c">#<Directory "/usr/lib/cgi-bin"></span>
<span class="c"># AllowOverride None</span>
<span class="c"># Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch</span>
<span class="c"># Require all granted</span>
<span class="c">#</Directory></span>
<span class="c">## cgi-bin config</span>
ScriptAlias /cgi-bin/ /var/www/cgi-bin/
<Directory <span class="s2">"/var/www/cgi-bin/"</span><span class="o">></span>
AllowOverride None
Options +ExecCGI
</Directory>
</IfDefine>
</IfModule>
<span class="c"># vim: syntax=apache ts=4 sw=4 sts=4 sr noet</span>
</code></pre></div></div>
<p>Now restart the <code class="language-plaintext highlighter-rouge">apache2</code></p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>service apache2 restart
</code></pre></div></div>
<h2 id="creating-a-simple-cgi-script">Creating a simple CGI script</h2>
<p>We have to first create the folder that will hold the cgi-scripts, so lets do that.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd</span> /var/www/cgi-bin
tasdik@Acer:/var/www/cgi-bin<span class="nv">$ </span><span class="nb">touch </span>hello.py
<span class="c">## make it executable for you and others</span>
tasdik@Acer:/var/www/cgi-bin<span class="nv">$ </span><span class="nb">chmod </span>o+x hello.py
<span class="c">## Put the following content just for testing inside `hello.py`</span>
tasdik@Acer:/var/www/cgi-bin<span class="nv">$ </span><span class="nb">sudo </span>nano hello.py
</code></pre></div></div>
<p>Add the following to <code class="language-plaintext highlighter-rouge">hello.py</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/env python
</span>
<span class="kn">import</span> <span class="nn">cgitb</span>
<span class="n">cgitb</span><span class="p">.</span><span class="n">enable</span><span class="p">()</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Content-Type: text/html;charset=utf-8"</span><span class="p">)</span>
<span class="k">print</span> <span class="s">"Content-type:text/html</span><span class="se">\r\n\r\n</span><span class="s">"</span>
<span class="k">print</span> <span class="s">'<html>'</span>
<span class="k">print</span> <span class="s">'<head>'</span>
<span class="k">print</span> <span class="s">'<title>Hello Word - First CGI Program</title>'</span>
<span class="k">print</span> <span class="s">'</head>'</span>
<span class="k">print</span> <span class="s">'<body>'</span>
<span class="k">print</span> <span class="s">'<h2>Hello Word! This is my first CGI program</h2>'</span>
<span class="k">print</span> <span class="s">'</body>'</span>
<span class="k">print</span> <span class="s">'</html>'</span>
</code></pre></div></div>
<h2 id="run-the-script">Run the Script:</h2>
<p>Open your browser and enter the following link</p>
<p><a href="http://localhost/cgi-bin/hello.py">http://localhost/cgi-bin/hello.py</a></p>
<p>And the script should run just fine.</p>
<h2 id="debugging-">Debugging :</h2>
<p>If the script is not running, you can check the logs stored in</p>
<p><code class="language-plaintext highlighter-rouge">/var/log/apache2/error.log</code></p>
<p>You can also refer the official reference here : <a href="http://httpd.apache.org/docs/2.0/howto/cgi.html">http://httpd.apache.org/docs/2.0/howto/cgi.html</a></p>
<p>Hope it helped!</p>
<h2 id="references">References:</h2>
<ul>
<li><a href="http://httpd.apache.org/docs/2.0/howto/cgi.html">http://httpd.apache.org/docs/2.0/howto/cgi.html</a></li>
<li><a href="http://askubuntu.com/questions/14763/where-are-the-apache-and-php-log-files">http://askubuntu.com/questions/14763/where-are-the-apache-and-php-log-files</a></li>
<li><a href="http://askubuntu.com/questions/679961/apache2-4-10-on-ubuntu-returning-internal-server-error-on-running-cgi-scripts/680039#680039">http://askubuntu.com/questions/679961/apache2-4-10-on-ubuntu-returning-internal-server-error-on-running-cgi-scripts/680039#680039</a></li>
</ul>
My Ramblings with Oracle-11g
2015-09-27T00:00:00+00:00
https://www.tasdikrahman.com/2015/09/27/My-Ramblings-about-Oracle-11g
<h2 id="intro-">Intro :</h2>
<p>I have recently started using <code class="language-plaintext highlighter-rouge">Oracle 11g</code> and boy, doesn’t it come with enough problems already!! However, it may be due to my undying allegiance to <code class="language-plaintext highlighter-rouge">mysql</code> and <code class="language-plaintext highlighter-rouge">sqlite</code>. But whatever be the case.</p>
<p>Here are some of the things which I found would be useful to a first time user of <code class="language-plaintext highlighter-rouge">Oracle 11g</code>. I will try to update it when I find something useful.</p>
<blockquote>
<p><strong>If you haven’t already installed <code class="language-plaintext highlighter-rouge">Oracle 11g</code>. <a href="http://www.tasdikrahman.com/2015/09/26/Install-and-Configure-Oracle-11g-on-Ubuntu-14.10/">I wrote a small article on how to do so sometime back</a>.</strong></p>
</blockquote>
<h2 id="creating-a-new-user-">Creating a new user :</h2>
<p>Now we can use the default users <code class="language-plaintext highlighter-rouge">SYSTEM</code> or <code class="language-plaintext highlighter-rouge">SYS</code>, but I prefer creating my own here.</p>
<p>To do that</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">tasdik</span><span class="o">@</span><span class="n">Acer</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> <span class="n">sqlplus</span>
<span class="k">SQL</span><span class="o">*</span><span class="n">Plus</span><span class="p">:</span> <span class="n">Release</span> <span class="mi">11</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span> <span class="n">Production</span> <span class="k">on</span> <span class="n">Sat</span> <span class="n">Sep</span> <span class="mi">26</span> <span class="mi">14</span><span class="p">:</span><span class="mi">25</span><span class="p">:</span><span class="mi">09</span> <span class="mi">2015</span>
<span class="n">Copyright</span> <span class="p">(</span><span class="k">c</span><span class="p">)</span> <span class="mi">1982</span><span class="p">,</span> <span class="mi">2011</span><span class="p">,</span> <span class="n">Oracle</span><span class="p">.</span> <span class="k">All</span> <span class="n">rights</span> <span class="n">reserved</span><span class="p">.</span>
<span class="n">Enter</span> <span class="k">user</span><span class="o">-</span><span class="n">name</span><span class="p">:</span> <span class="k">SYSTEM</span>
<span class="n">Enter</span> <span class="n">password</span><span class="p">:</span>
<span class="n">Connected</span> <span class="k">to</span><span class="p">:</span>
<span class="n">Oracle</span> <span class="k">Database</span> <span class="mi">11</span><span class="k">g</span> <span class="n">Express</span> <span class="n">Edition</span> <span class="n">Release</span> <span class="mi">11</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span> <span class="o">-</span> <span class="mi">64</span><span class="nb">bit</span> <span class="n">Production</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">create</span> <span class="k">user</span> <span class="n">lab</span> <span class="n">identified</span> <span class="k">by</span> <span class="n">lab</span><span class="p">;</span>
<span class="k">User</span> <span class="n">created</span><span class="p">.</span>
<span class="o">##</span> <span class="k">check</span> <span class="n">the</span> <span class="k">new</span> <span class="k">user</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">select</span> <span class="n">username</span> <span class="k">from</span> <span class="n">dba_users</span> <span class="p">;</span>
<span class="n">USERNAME</span>
<span class="c1">------------------------------</span>
<span class="n">LAB</span>
<span class="n">SYS</span>
<span class="k">SYSTEM</span>
<span class="n">ANONYMOUS</span>
<span class="n">APEX_PUBLIC_USER</span>
<span class="n">APEX_040000</span>
<span class="n">OUTLN</span>
<span class="n">XS</span><span class="err">$</span><span class="k">NULL</span>
<span class="n">FLOWS_FILES</span>
<span class="n">MDSYS</span>
<span class="n">USERNAME</span>
<span class="c1">------------------------------</span>
<span class="n">CTXSYS</span>
<span class="n">XDB</span>
<span class="n">HR</span>
<span class="mi">14</span> <span class="k">rows</span> <span class="n">selected</span><span class="p">.</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">grant</span> <span class="k">create</span> <span class="k">session</span> <span class="k">to</span> <span class="n">lab</span> <span class="p">;</span>
<span class="k">Grant</span> <span class="n">succeeded</span><span class="p">.</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">grant</span> <span class="k">connect</span> <span class="k">to</span> <span class="n">lab</span> <span class="p">;</span>
<span class="k">Grant</span> <span class="n">succeeded</span><span class="p">.</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">grant</span> <span class="k">all</span> <span class="k">privileges</span> <span class="k">to</span> <span class="n">lab</span> <span class="p">;</span>
<span class="k">Grant</span> <span class="n">succeeded</span><span class="p">.</span>
<span class="k">SQL</span><span class="o">></span> <span class="n">exit</span>
<span class="n">Disconnected</span> <span class="k">from</span> <span class="n">Oracle</span> <span class="k">Database</span> <span class="mi">11</span><span class="k">g</span> <span class="n">Express</span> <span class="n">Edition</span> <span class="n">Release</span> <span class="mi">11</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span> <span class="o">-</span> <span class="mi">64</span><span class="nb">bit</span> <span class="n">Production</span></code></pre></figure>
<p>Now connect to the new user <code class="language-plaintext highlighter-rouge">lab</code></p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">tasdik</span><span class="o">@</span><span class="n">Acer</span><span class="p">:</span><span class="o">~</span><span class="err">$</span> <span class="n">sqlplus</span>
<span class="k">SQL</span><span class="o">*</span><span class="n">Plus</span><span class="p">:</span> <span class="n">Release</span> <span class="mi">11</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span> <span class="n">Production</span> <span class="k">on</span> <span class="n">Sat</span> <span class="n">Sep</span> <span class="mi">26</span> <span class="mi">14</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">10</span> <span class="mi">2015</span>
<span class="n">Copyright</span> <span class="p">(</span><span class="k">c</span><span class="p">)</span> <span class="mi">1982</span><span class="p">,</span> <span class="mi">2011</span><span class="p">,</span> <span class="n">Oracle</span><span class="p">.</span> <span class="k">All</span> <span class="n">rights</span> <span class="n">reserved</span><span class="p">.</span>
<span class="n">Enter</span> <span class="k">user</span><span class="o">-</span><span class="n">name</span><span class="p">:</span> <span class="n">lab</span>
<span class="n">Enter</span> <span class="n">password</span><span class="p">:</span>
<span class="n">Connected</span> <span class="k">to</span><span class="p">:</span>
<span class="n">Oracle</span> <span class="k">Database</span> <span class="mi">11</span><span class="k">g</span> <span class="n">Express</span> <span class="n">Edition</span> <span class="n">Release</span> <span class="mi">11</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span> <span class="o">-</span> <span class="mi">64</span><span class="nb">bit</span> <span class="n">Production</span>
<span class="o">##</span> <span class="n">clear</span> <span class="n">the</span> <span class="n">screen</span>
<span class="k">SQL</span><span class="o">></span> <span class="n">cl</span> <span class="n">scr</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">select</span> <span class="k">table_name</span> <span class="k">from</span> <span class="n">user_tables</span> <span class="p">;</span>
<span class="k">no</span> <span class="k">rows</span> <span class="n">selected</span>
<span class="o">##</span> <span class="k">as</span> <span class="k">no</span> <span class="n">tables</span> <span class="n">have</span> <span class="n">been</span> <span class="n">created</span> <span class="k">by</span> <span class="n">the</span> <span class="k">user</span> <span class="nv">`lab`</span> <span class="n">till</span> <span class="n">now</span>
<span class="o">##</span> <span class="n">so</span> <span class="n">lets</span> <span class="k">create</span> <span class="k">some</span><span class="p">,</span> <span class="n">shall</span> <span class="n">we</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">department</span> <span class="p">(</span>
<span class="mi">2</span> <span class="n">dept_name</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">20</span><span class="p">),</span>
<span class="mi">3</span> <span class="n">building</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">15</span><span class="p">),</span>
<span class="mi">4</span> <span class="n">budget</span> <span class="nb">numeric</span><span class="p">(</span><span class="mi">12</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span> <span class="k">check</span> <span class="p">(</span><span class="n">budget</span> <span class="o">></span> <span class="mi">0</span><span class="p">),</span>
<span class="mi">5</span> <span class="k">primary</span> <span class="k">key</span> <span class="p">(</span><span class="n">dept_name</span><span class="p">)</span>
<span class="mi">6</span> <span class="p">)</span> <span class="p">;</span>
<span class="k">Table</span> <span class="n">created</span><span class="p">.</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">select</span> <span class="k">table_name</span> <span class="k">from</span> <span class="n">user_tables</span> <span class="p">;</span>
<span class="k">TABLE_NAME</span>
<span class="c1">------------------------------</span>
<span class="n">DEPARTMENT</span>
<span class="o">##</span> <span class="n">you</span> <span class="n">can</span> <span class="k">do</span> <span class="n">whatever</span> <span class="n">you</span> <span class="n">can</span> <span class="n">normally</span> <span class="k">do</span> <span class="k">with</span> <span class="n">this</span> <span class="k">user</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">drop</span> <span class="k">table</span> <span class="n">department</span> <span class="p">;</span>
<span class="k">Table</span> <span class="n">dropped</span><span class="p">.</span>
<span class="k">SQL</span><span class="o">></span> <span class="k">select</span> <span class="k">table_name</span> <span class="k">from</span> <span class="n">user_tables</span> <span class="p">;</span>
<span class="k">no</span> <span class="k">rows</span> <span class="n">selected</span>
<span class="k">SQL</span><span class="o">></span> </code></pre></figure>
<h2 id="to-delete-a-user-">To Delete a User :</h2>
<p>After dropping the user, you need to, for each related tablespace, take it offline and drop it. For example if you had a user named ‘SAMPLE’ and two tablespaces called ‘SAMPLE’ and ‘SAMPLE_INDEX’, then you’d need to do the following:</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="code"><pre><span class="k">DROP</span> <span class="k">USER</span> <span class="n">SAMPLE</span> <span class="k">CASCADE</span><span class="p">;</span>
<span class="k">ALTER</span> <span class="n">TABLESPACE</span> <span class="n">SAMPLE</span> <span class="n">OFFLINE</span><span class="p">;</span>
<span class="k">DROP</span> <span class="n">TABLESPACE</span> <span class="n">SAMPLE</span> <span class="k">INCLUDING</span> <span class="n">CONTENTS</span><span class="p">;</span>
<span class="k">ALTER</span> <span class="n">TABLESPACE</span> <span class="n">SAMPLE_INDEX</span> <span class="n">OFFLINE</span><span class="p">;</span>
<span class="k">DROP</span> <span class="n">TABLESPACE</span> <span class="n">SAMPLE_INDEX</span> <span class="k">INCLUDING</span> <span class="n">CONTENTS</span><span class="p">;</span>
</pre></td></tr></tbody></table></code></pre></figure>
<h2 id="there-is-no-such-thing-as---if-exists--">There is no such thing as <code class="language-plaintext highlighter-rouge">IF EXISTS</code> :</h2>
<p>I mean what! I find this really irritating in the part of <code class="language-plaintext highlighter-rouge">Oracle</code> to not give support for this feature as <strong>any other major RDMS system</strong> implements this.</p>
<p>Lets try it out</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SQL</span><span class="o">></span> <span class="k">select</span> <span class="k">table_name</span> <span class="k">from</span> <span class="n">user_tables</span> <span class="p">;</span>
<span class="k">no</span> <span class="k">rows</span> <span class="n">selected</span>
<span class="k">SQL</span><span class="o">></span></code></pre></figure>
<p>So we don’t have any <code class="language-plaintext highlighter-rouge">relations</code> right now. Lets try to drop <code class="language-plaintext highlighter-rouge">tasdik</code> which does not exist the way we do in <code class="language-plaintext highlighter-rouge">mysql</code>.</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SQL</span><span class="o">></span> <span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">tasdik</span> <span class="p">;</span>
<span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">tasdik</span>
<span class="o">*</span>
<span class="n">ERROR</span> <span class="k">at</span> <span class="n">line</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">ORA</span><span class="o">-</span><span class="mi">00933</span><span class="p">:</span> <span class="k">SQL</span> <span class="n">command</span> <span class="k">not</span> <span class="n">properly</span> <span class="n">ended</span>
<span class="k">SQL</span><span class="o">></span></code></pre></figure>
<p>If you try to <code class="language-plaintext highlighter-rouge">DROP</code> a relation which does not exist, the query will not stop executing in the middle, but will just give you an error like the above</p>
<p>Well, <code class="language-plaintext highlighter-rouge">sqlite</code> has a limitation where we cannot <code class="language-plaintext highlighter-rouge">alter</code> the <code class="language-plaintext highlighter-rouge">attribute</code> in a <code class="language-plaintext highlighter-rouge">relation</code>, like <code class="language-plaintext highlighter-rouge">add</code> or <code class="language-plaintext highlighter-rouge">delete</code> an attribute to a relation. So nobody is perfect here.</p>
<h2 id="no-support-for-cursor-keys-">No support for cursor keys :</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span>sqlplus
SQL<span class="k">*</span>Plus: Release 11.2.0.2.0 Production on Sun Sep 27 19:12:41 2015
Copyright <span class="o">(</span>c<span class="o">)</span> 1982, 2011, Oracle. All rights reserved.
Enter user-name: lab
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> ^[[A^[[A^[[A</code></pre></figure>
<p>As you can see, whenever you press the cursor keys, there is garbage on the screen. Now I was not sure why this was happening. Turns out it doesn’t support the arrow keys in <code class="language-plaintext highlighter-rouge">*nix</code> based systems. (Good news for those on <code class="language-plaintext highlighter-rouge">windows</code>).</p>
<p>But well there is a workaround.</p>
<p>You can use a third party utility called <code class="language-plaintext highlighter-rouge">rlwrap</code>.</p>
<blockquote>
<p>rlwrap is a readline wrapper, a small utility that uses the GNU readline library to allow the editing of keyboard input for any other command. It maintains a separate input history for each command, and can TAB-expand words using all previously seen words and/or a user-specified file.So you will be able to use arrows and also get a command history as a bonus.</p>
</blockquote>
<p>After you have installed the utility run sqlplus the following way:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span>rlwrap sqlplus
SQL<span class="k">*</span>Plus: Release 11.2.0.2.0 Production on Sun Sep 27 19:04:38 2015
Copyright <span class="o">(</span>c<span class="o">)</span> 1982, 2011, Oracle. All rights reserved.
Enter user-name: lab
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL></code></pre></figure>
<p>And the arrow keys would work just fine</p>
<h2 id="references">References:</h2>
<ul>
<li><a href="http://stackoverflow.com/questions/22386976/create-a-user-with-all-privileges-in-oracle">http://stackoverflow.com/questions/22386976/create-a-user-with-all-privileges-in-oracle</a></li>
<li><a href="http://stackoverflow.com/questions/205736/get-list-of-all-tables-in-oracle">http://stackoverflow.com/questions/205736/get-list-of-all-tables-in-oracle</a></li>
<li><a href="http://stackoverflow.com/questions/969245/how-to-delete-a-user-in-oracle-10-including-all-its-tablespace-and-datafiles">http://stackoverflow.com/questions/969245/how-to-delete-a-user-in-oracle-10-including-all-its-tablespace-and-datafiles</a></li>
<li><a href="http://stackoverflow.com/questions/9890636/arrow-keys-are-not-functional-in-sqlplus">http://stackoverflow.com/questions/9890636/arrow-keys-are-not-functional-in-sqlplus</a></li>
</ul>
Install and Configure Oracle-11g on Ubuntu-14.10
2015-09-26T00:00:00+00:00
https://www.tasdikrahman.com/2015/09/26/Install-and-Configure-Oracle-11g-on-Ubuntu-14.10
<h2 id="intro-">Intro :</h2>
<p>Well as luck would have it, we were having our Database labs in <code class="language-plaintext highlighter-rouge">ORACLE 11g</code> as part of the coursework. And I must admit, I still have my love for <code class="language-plaintext highlighter-rouge">mysql</code>.</p>
<p>I installed it on my system(after a lot of trials!) this way.</p>
<blockquote>
<p><a href="http://www.tasdikrahman.com/2015/09/27/My-Ramblings-about-Oracle-11g/">Next part of this article</a></p>
</blockquote>
<h2 id="pre-requisites">Pre-requisites:</h2>
<p>JAVA should be installed and the environment variable for it should be set. Check it by doing</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span>java <span class="nt">-version</span>
java version <span class="s2">"1.7.0_79"</span>
OpenJDK Runtime Environment <span class="o">(</span>IcedTea 2.5.5<span class="o">)</span> <span class="o">(</span>7u79-2.5.5-0ubuntu0.14.10.2<span class="o">)</span>
OpenJDK 64-Bit Server VM <span class="o">(</span>build 24.79-b02, mixed mode<span class="o">)</span>
tasdik@Acer:~<span class="nv">$ </span><span class="nb">echo</span> <span class="nv">$JAVA_HOME</span>
/usr/lib/jvm/java-7-openjdk-amd64
tasdik@Acer:~<span class="nv">$ </span></code></pre></figure>
<p>If not you can set it by placing the installed <code class="language-plaintext highlighter-rouge">jvm</code> to <code class="language-plaintext highlighter-rouge">/etc/environment</code></p>
<p>Mine looks like this</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="code"><pre>tasdik@Acer:~<span class="nv">$ </span><span class="nb">cat</span> /etc/environment
<span class="nv">PATH</span><span class="o">=</span><span class="s2">"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"</span>
<span class="nv">http_proxy</span><span class="o">=</span><span class="s2">"http://172.16.0.16:8080/"</span>
<span class="nv">https_proxy</span><span class="o">=</span><span class="s2">"https://172.16.0.16:8080/"</span>
<span class="nv">ftp_proxy</span><span class="o">=</span><span class="s2">"ftp://172.16.0.16:8080/"</span>
<span class="nv">socks_proxy</span><span class="o">=</span><span class="s2">"socks://172.16.0.16:8080/"</span>
<span class="nv">JAVA_HOME</span><span class="o">=</span><span class="s2">"/usr/lib/jvm/java-7-openjdk-amd64"</span>
tasdik@Acer:~<span class="nv">$ </span>
</pre></td></tr></tbody></table></code></pre></figure>
<h2 id="installing-oracle-11g-r2-express-edition">Installing Oracle 11g R2 Express Edition</h2>
<p>A couple of extra things are needed</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>alien libaio1 unixodbc</code></pre></figure>
<p>Download Oracle 11g, you need to make an account for that(Yeah, I know!)</p>
<p>Link : <a href="http://www.oracle.com/technetwork/database/database-technologies/express-edition/downloads/index.html">http://www.oracle.com/technetwork/database/database-technologies/express-edition/downloads/index.html</a></p>
<p>Place annd Unzip the <code class="language-plaintext highlighter-rouge">zip </code> file to the Directory of your choice. I did mine to ‘Documents’</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd </span>Documents/
tasdik@Acer:~/Documents<span class="nv">$ </span>unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip
<span class="c">## A new directory was added</span>
tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd </span>Documents/Disk1/</code></pre></figure>
<p>Now we need to convert the rpm (A red hat package to .deb) package</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">sudo </span>alien <span class="nt">--scripts</span> <span class="nt">-d</span> oracle-xe-11.2.0-1.0.x86_64.rpm</code></pre></figure>
<p>This will take a while, so open another <code class="language-plaintext highlighter-rouge">terminal</code> window</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">sudo </span>nano /sbin/chkconfig</code></pre></figure>
<p>And then add the following to it</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
</pre></td><td class="code"><pre><span class="c">#!/bin/bash</span>
<span class="c"># Oracle 11gR2 XE installer chkconfig hack for Ubuntu</span>
<span class="nv">file</span><span class="o">=</span>/etc/init.d/oracle-xe
<span class="k">if</span> <span class="o">[[</span> <span class="o">!</span> <span class="sb">`</span><span class="nb">tail</span> <span class="nt">-n1</span> <span class="nv">$file</span> | <span class="nb">grep </span>INIT<span class="sb">`</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'### BEGIN INIT INFO'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'# Provides: OracleXE'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'# Required-Start: $remote_fs $syslog'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'# Required-Stop: $remote_fs $syslog'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'# Default-Start: 2 3 4 5'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'# Default-Stop: 0 1 6'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'# Short-Description: Oracle 11g Express Edition'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="nb">echo</span> <span class="s1">'### END INIT INFO'</span> <span class="o">>></span> <span class="nv">$file</span>
<span class="k">fi
</span>update-rc.d oracle-xe defaults 80 01
<span class="c">#EOF</span>
</pre></td></tr></tbody></table></code></pre></figure>
<p>Save it and give the required execution privileges</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo chmod </span>755 /sbin/chkconfig</code></pre></figure>
<p>After this, we have to create the file <code class="language-plaintext highlighter-rouge">/etc/sysctl.d/60-oracle.conf</code> to set the additional kernel parameters. Open the file by executing the following statement.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/sysctl.d/60-oracle.conf</code></pre></figure>
<p>Copy and paste the following into the file. Kernel.shmmax is the maximum possible value of physical RAM in bytes. <code class="language-plaintext highlighter-rouge">536870912 / 1024 /1024 = 512 MB.</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="code"><pre><span class="c"># Oracle 11g XE kernel parameters</span>
fs.file-max<span class="o">=</span>6815744
net.ipv4.ip_local_port_range<span class="o">=</span>9000 65000
kernel.sem<span class="o">=</span>250 32000 100 128
kernel.shmmax<span class="o">=</span>536870912
</pre></td></tr></tbody></table></code></pre></figure>
<p>Load the kernel parameters:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">sudo </span>service procps start</code></pre></figure>
<p>The changes may be verified again by executing:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">sudo </span>sysctl <span class="nt">-q</span> fs.file-max
fs.file-max <span class="o">=</span> 6815744
tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span></code></pre></figure>
<p>After this, execute the following statements to make some more required changes:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">sudo ln</span> <span class="nt">-s</span> /usr/bin/awk /bin/awk
tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">mkdir</span> /var/lock/subsys
tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">touch</span> /var/lock/subsys/listener</code></pre></figure>
<h2 id="install-the-deb-file">Install the .deb file</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/Documents/Disk1<span class="nv">$ </span><span class="nb">sudo </span>dpkg <span class="nt">--install</span> oracle-xe_11.2.0-2_amd64.deb</code></pre></figure>
<p>After that,
Execute the following to avoid getting a ORA-00845: MEMORY_TARGET error. Note: replace “size=3804m” with the size of your (virtual) machine’s RAM in MBs.</p>
<blockquote>
<p>Note : Close Chrome before executing the following three lines as it crashes when doing so</p>
</blockquote>
<p>To check your system RAM (allated), do a</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span>free <span class="nt">-tom</span>
total used free shared buffers cached
Mem: 3804 3465 338 910 81 1642
Swap: 3943 30 3913
Total: 7748 3496 4252
tasdik@Acer:~<span class="nv">$ </span></code></pre></figure>
<p>My RAM is <code class="language-plaintext highlighter-rouge">3804</code> here
Then do the following</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo rm</span> <span class="nt">-rf</span> /dev/shm
tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo mkdir</span> /dev/shm
tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>mount <span class="nt">-t</span> tmpfs shmfs <span class="nt">-o</span> <span class="nv">size</span><span class="o">=</span>3804m /dev/shm</code></pre></figure>
<p>Create the file /etc/rc2.d/S01shm_load.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/rc2.d/S01shm_load</code></pre></figure>
<p>And then add the following</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="code"><pre><span class="c">#!/bin/sh</span>
<span class="k">case</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="k">in
</span>start<span class="p">)</span> <span class="nb">mkdir</span> /var/lock/subsys 2>/dev/null
<span class="nb">touch</span> /var/lock/subsys/listener
<span class="nb">rm</span> /dev/shm 2>/dev/null
<span class="nb">mkdir</span> /dev/shm 2>/dev/null
mount <span class="nt">-t</span> tmpfs shmfs <span class="nt">-o</span> <span class="nv">size</span><span class="o">=</span>3804m /dev/shm <span class="p">;;</span>
<span class="k">*</span><span class="p">)</span> <span class="nb">echo </span>error
<span class="nb">exit </span>1 <span class="p">;;</span>
<span class="k">esac</span>
</pre></td></tr></tbody></table></code></pre></figure>
<p>Give permissions to it</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo chmod </span>755 /etc/rc2.d/S01shm_load</code></pre></figure>
<h2 id="configuring-oracle-11g-r2-express-edition">Configuring Oracle 11g R2 Express Edition</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo</span> /etc/init.d/oracle-xe configure</code></pre></figure>
<p>Go on choosing the defaults. I chose, not to start <code class="language-plaintext highlighter-rouge">Oracle on startup</code>. So do accordingly.</p>
<p>Setting the environment vars</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>gedit /etc/bash.bashrc</code></pre></figure>
<p>And add the following</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="code"><pre><span class="c">#### for oracle 11g</span>
<span class="nb">export </span><span class="nv">ORACLE_HOME</span><span class="o">=</span>/u01/app/oracle/product/11.2.0/xe
<span class="nb">export </span><span class="nv">ORACLE_SID</span><span class="o">=</span>XE
<span class="nb">export </span><span class="nv">NLS_LANG</span><span class="o">=</span><span class="sb">`</span><span class="nv">$ORACLE_HOME</span>/bin/nls_lang.sh<span class="sb">`</span>
<span class="nb">export </span><span class="nv">ORACLE_BASE</span><span class="o">=</span>/u01/app/oracle
<span class="nb">export </span><span class="nv">LD_LIBRARY_PATH</span><span class="o">=</span><span class="nv">$ORACLE_HOME</span>/lib:<span class="nv">$LD_LIBRARY_PATH</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="nv">$ORACLE_HOME</span>/bin:<span class="nv">$PATH</span>
</pre></td></tr></tbody></table></code></pre></figure>
<p>Save it and <code class="language-plaintext highlighter-rouge">source</code> it</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">source</span> /etc/bash.bashrc
<span class="c">## check the environment variables</span>
tasdik@Acer:~<span class="nv">$ </span><span class="nb">echo</span> <span class="nv">$ORACLE_HOME</span>
/u01/app/oracle/product/11.2.0/xe
tasdik@Acer:~<span class="nv">$ </span></code></pre></figure>
<p>I recommend rebooting the system at this point of time</p>
<h2 id="after-a-system-reboot">After a System Reboot</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>service oracle-xe start</code></pre></figure>
<p>A file named oraclexe-gettingstarted.desktop is placed on your desktop. To make this file executable, navigate to you desktop.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd</span> ~/Desktop
tasdik@Acer:~/Desktop<span class="nv">$ </span><span class="nb">sudo chmod </span>a+x oraclexe-gettingstarted.desktop</code></pre></figure>
<h2 id="running-sqlplus">Running <code class="language-plaintext highlighter-rouge">SQlplus</code></h2>
<p>You have to Unset <code class="language-plaintext highlighter-rouge">http_proxy</code> and <code class="language-plaintext highlighter-rouge">no_proxy</code>.</p>
<p>To do that</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">unset </span>http_proxy
tasdik@Acer:~<span class="nv">$ </span><span class="nb">unset </span>no_proxy
tasdik@Acer:~<span class="nv">$ </span>sqlplus
SQL<span class="k">*</span>Plus: Release 11.2.0.2.0 Production on Sat Sep 26 11:34:06 2015
Copyright <span class="o">(</span>c<span class="o">)</span> 1982, 2011, Oracle. All rights reserved.
Enter user-name: SYSTEM
Enter password:
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL> </code></pre></figure>
<p>There you go!</p>
<h2 id="ending-notes">Ending notes:</h2>
<p>Remember to do a</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>service oracle-xe start</code></pre></figure>
<p>at startup, if you chose not to Start <code class="language-plaintext highlighter-rouge">Oracle 11 g</code> at system startup</p>
Workaround for deleting Columns in SQLite
2015-09-18T00:00:00+00:00
https://www.tasdikrahman.com/2015/09/18/Workaround-to-Delete-column-in-SQLite-Database
<h2 id="intro-">Intro :</h2>
<p>SQLite has limited <code class="language-plaintext highlighter-rouge">ALTER TABLE</code> support that you can use to add a column to the end of a table or to change the name of a table. If you want to make more complex changes in the structure of a table, you will have to recreate the table. You can save existing data to a temporary table, drop the old table, create the new table, then copy the data back in from the temporary table.</p>
<p>From the docs.</p>
<blockquote>
<p>It is not possible to rename a column, remove a column, or add or remove constraints from a table.</p>
</blockquote>
<p>source : <a href="http://www.sqlite.org/lang_altertable.html">http://www.sqlite.org/lang_altertable.html</a></p>
<p>While you can always create a new table and then drop the older one.</p>
<p>I will try to explain this <a href="http://www.sqlite.org/faq.html#q11">this workaround </a> with an example.</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">sqlite</span><span class="o">></span> <span class="p">.</span><span class="k">schema</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">person</span><span class="p">(</span>
<span class="n">id</span> <span class="nb">INTEGER</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="n">first_name</span> <span class="nb">TEXT</span><span class="p">,</span>
<span class="n">last_name</span> <span class="nb">TEXT</span><span class="p">,</span>
<span class="n">age</span> <span class="nb">INTEGER</span><span class="p">,</span>
<span class="n">height</span> <span class="nb">INTEGER</span>
<span class="p">);</span>
<span class="n">sqlite</span><span class="o">></span> <span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">person</span> <span class="p">;</span>
<span class="n">id</span> <span class="n">first_name</span> <span class="n">last_name</span> <span class="n">age</span> <span class="n">height</span>
<span class="c1">---------- ---------- ---------- ---------- ----------</span>
<span class="mi">0</span> <span class="n">john</span> <span class="n">doe</span> <span class="mi">20</span> <span class="mi">170</span>
<span class="mi">1</span> <span class="n">foo</span> <span class="n">bar</span> <span class="mi">25</span> <span class="mi">171</span> </code></pre></figure>
<p>Now you want to remove the column <code class="language-plaintext highlighter-rouge">height</code> from this table.</p>
<p>Create another table called <code class="language-plaintext highlighter-rouge">new_person</code></p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">sqlite</span><span class="o">></span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">new_person</span><span class="p">(</span>
<span class="p">...</span><span class="o">></span> <span class="n">id</span> <span class="nb">INTEGER</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="p">...</span><span class="o">></span> <span class="n">first_name</span> <span class="nb">TEXT</span><span class="p">,</span>
<span class="p">...</span><span class="o">></span> <span class="n">last_name</span> <span class="nb">TEXT</span><span class="p">,</span>
<span class="p">...</span><span class="o">></span> <span class="n">age</span> <span class="nb">INTEGER</span>
<span class="p">...</span><span class="o">></span> <span class="p">)</span> <span class="p">;</span>
<span class="n">sqlite</span><span class="o">></span> </code></pre></figure>
<p>Now copy the data from the old table</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">sqlite</span><span class="o">></span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">new_person</span>
<span class="p">...</span><span class="o">></span> <span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span> <span class="n">first_name</span><span class="p">,</span> <span class="n">last_name</span><span class="p">,</span> <span class="n">age</span> <span class="k">FROM</span> <span class="n">person</span> <span class="p">;</span>
<span class="n">sqlite</span><span class="o">></span> <span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">new_person</span> <span class="p">;</span>
<span class="n">id</span> <span class="n">first_name</span> <span class="n">last_name</span> <span class="n">age</span>
<span class="c1">---------- ---------- ---------- ----------</span>
<span class="mi">0</span> <span class="n">john</span> <span class="n">doe</span> <span class="mi">20</span>
<span class="mi">1</span> <span class="n">foo</span> <span class="n">bar</span> <span class="mi">25</span>
<span class="n">sqlite</span><span class="o">></span></code></pre></figure>
<p>Now Drop the <code class="language-plaintext highlighter-rouge">person</code> table and rename <code class="language-plaintext highlighter-rouge">new_person</code> to <code class="language-plaintext highlighter-rouge">person</code></p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">sqlite</span><span class="o">></span> <span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">person</span> <span class="p">;</span>
<span class="n">sqlite</span><span class="o">></span> <span class="k">ALTER</span> <span class="k">TABLE</span> <span class="n">new_person</span> <span class="k">RENAME</span> <span class="k">TO</span> <span class="n">person</span> <span class="p">;</span>
<span class="n">sqlite</span><span class="o">></span></code></pre></figure>
<p>So now if you do a <code class="language-plaintext highlighter-rouge">.schema</code>, you will see</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="n">sqlite</span><span class="o">></span><span class="p">.</span><span class="k">schema</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="nv">"person"</span><span class="p">(</span>
<span class="n">id</span> <span class="nb">INTEGER</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="n">first_name</span> <span class="nb">TEXT</span><span class="p">,</span>
<span class="n">last_name</span> <span class="nb">TEXT</span><span class="p">,</span>
<span class="n">age</span> <span class="nb">INTEGER</span>
<span class="p">);</span></code></pre></figure>
<p>So that’s how you insert and delete columns from an SQLite Database</p>
Install Hadoop(Multi Node)
2015-09-13T00:00:00+00:00
https://www.tasdikrahman.com/2015/09/13/Setup-Hadoop-on-Ubuntu-Multi-Node-Cluster
<h2 id="intro">Intro:</h2>
<p>In this article, I will describe the required steps for setting up a distributed, multi-node Apache Hadoop cluster backed by the Hadoop Distributed File System (HDFS), running on Ubuntu Linux.</p>
<p>I am using <code class="language-plaintext highlighter-rouge">nano</code> as the text editor for this article but you can use any other text-editor like <code class="language-plaintext highlighter-rouge">vi</code>, <code class="language-plaintext highlighter-rouge">sublime</code> or <code class="language-plaintext highlighter-rouge">atom</code> for doing the same</p>
<blockquote>
<p>I suggest you set up hadoop on single node before directly going for multi node as it will be easier to debug any errors. I have explained about <a href="http://www.tasdikrahman.com/2015/09/10/Install-Hadoop-1.0.3-Single-Node-on-Ubuntu-14.04.2/">how to setup single node hadoop on ubuntu</a></p>
</blockquote>
<p><strong>Note</strong> :</p>
<p>I will be demonstrating for 3 nodes but you can add more nodes as you like.</p>
<h2 id="installation">Installation</h2>
<blockquote>
<p>Install for the three node <a href="http://www.tasdikrahman.com/2015/09/10/Install-Hadoop-1.0.3-Single-Node-on-Ubuntu-14.04.2/">using this tutuorial</a></p>
</blockquote>
<h2 id="done-lets-continue-then">Done? Let’s continue then!</h2>
<p>The very first step would be stop <code class="language-plaintext highlighter-rouge">hadoop</code> on all the three machines.</p>
<p>For that, you just need to do</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys9:~/hadoop<span class="nv">$ </span>bin/stop-all.sh</code></pre></figure>
<p>on each of the nodes</p>
<p>Now to check whether hadoop processes are stopped in all the nodes or not!</p>
<p><strong>For sys9</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys9:~/hadoop<span class="nv">$ </span>jps
12792 Jps
hadoop@sys9:~/hadoop<span class="err">$</span></code></pre></figure>
<p><strong>For sys10</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys10:~/hadoop<span class="nv">$ </span>jps
12637 Jps
hadoop@sys10:~/hadoop<span class="err">$</span></code></pre></figure>
<p><strong>For sys8</strong></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span>jps
2186 Jps
hadoop@sys8:~<span class="nv">$ </span></code></pre></figure>
<p>Now we have to choose which one node will be the master node, so that the other two nodes can be the slaves</p>
<p>In my case, I will chose <code class="language-plaintext highlighter-rouge">sys8</code> to be my <code class="language-plaintext highlighter-rouge">master</code> node and the others to be <code class="language-plaintext highlighter-rouge">slaves</code></p>
<h2 id="master-node-configuration-">Master node Configuration :</h2>
<p>Add the the ip address of the slaves to <code class="language-plaintext highlighter-rouge">/etc/hosts</code> on <code class="language-plaintext highlighter-rouge">sys8</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/hosts</code></pre></figure>
<p>It should look something like this after you have added it.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">### master node</span>
hadoop@sys8:~<span class="nv">$ </span><span class="nb">cat</span> /etc/hosts
192.168.103.26 sys9 <span class="c">## slave</span>
192.168.103.28 sys10 <span class="c">## slave</span>
hadoop@sys8:~<span class="nv">$ </span></code></pre></figure>
<p>For <code class="language-plaintext highlighter-rouge">sys9</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">### for slave node</span>
hadoop@sys9:~<span class="nv">$ </span><span class="nb">cat</span> /etc/hosts
192.168.103.24 sys8 <span class="c">## master</span>
192.168.103.26 sys9 <span class="c">## slave</span>
192.168.103.28 sys10 <span class="c">## slave</span>
hadoop@sys9:~<span class="nv">$ </span></code></pre></figure>
<p>For <code class="language-plaintext highlighter-rouge">sys10</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">### for slave node</span>
hadoop@sys10:~<span class="nv">$ </span><span class="nb">cat</span> /etc/hosts
192.168.103.24 sys8 <span class="c">## master</span>
192.168.103.26 sys9 <span class="c">## slave</span>
192.168.103.28 sys10 <span class="c">## slave</span>
hadoop@sys10:~<span class="err">$</span></code></pre></figure>
<h2 id="changing-the-hadoop-configurations-">Changing the hadoop configurations :</h2>
<p>Shift to the <code class="language-plaintext highlighter-rouge">hadoop</code> directory first</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">cd</span> ~/hadoop/conf
hadoop@sys8:~/hadoop/conf<span class="nv">$ </span> </code></pre></figure>
<h3 id="editing-confcore-sitexml-all-nodes">Editing <code class="language-plaintext highlighter-rouge">conf/core-site.xml</code> (all nodes)</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span><span class="nb">sudo </span>nano core-site.xml </code></pre></figure>
<p>After editing, it should look like</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
</pre></td><td class="code"><pre><span class="cp"><?xml version="1.0"?></span>
<span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span>
<span class="c"><!-- Put site-specific property overrides in this file. --></span>
<span class="nt"><configuration></span>
<span class="nt"><property></span>
<span class="nt"><name></span>hadoop.tmp.dir<span class="nt"></name></span>
<span class="nt"><value></span>/app/hadoop/tmp<span class="nt"></value></span>
<span class="nt"><description></span>A base for other temporary directories.<span class="nt"></description></span>
<span class="nt"></property></span>
<span class="nt"><property></span>
<span class="nt"><name></span>fs.default.name<span class="nt"></name></span>
<span class="nt"><value></span>hdfs://192.168.103.24:54310<span class="nt"></value></span>
<span class="nt"><description></span>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.<span class="nt"></description></span>
<span class="nt"></property></span>
<span class="nt"></configuration></span>
</pre></td></tr></tbody></table></code></pre></figure>
<p>similarly for nodes <code class="language-plaintext highlighter-rouge">sys9</code> and <code class="language-plaintext highlighter-rouge">sys10</code>.</p>
<p><strong>Note</strong></p>
<p>Here <code class="language-plaintext highlighter-rouge">192.168.103.24</code> is the ip address of the master node.
We just have replaced <code class="language-plaintext highlighter-rouge">localhost</code> with this ip address</p>
<h3 id="editing-confmapred-sitexml-all-nodes">Editing <code class="language-plaintext highlighter-rouge">conf/mapred-site.xml</code> (All nodes)</h3>
<p>After editing, the file should be</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="code"><pre><span class="cp"><?xml version="1.0"?></span>
<span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span>
<span class="c"><!-- Put site-specific property overrides in this file. --></span>
<span class="nt"><configuration></span>
<span class="nt"><property></span>
<span class="nt"><name></span>mapred.job.tracker<span class="nt"></name></span>
<span class="nt"><value></span>192.168.103.24:54311<span class="nt"></value></span>
<span class="nt"><description></span>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
<span class="nt"></description></span>
<span class="nt"></property></span>
<span class="nt"></configuration></span>
</pre></td></tr></tbody></table></code></pre></figure>
<p>similarly for nodes <code class="language-plaintext highlighter-rouge">sys9</code> and <code class="language-plaintext highlighter-rouge">sys10</code>.</p>
<p><strong>Note</strong> :</p>
<p>The file <code class="language-plaintext highlighter-rouge">conf/hdfs-site.xml</code> remains the same for all the nodes</p>
<h3 id="editing-the-confmasters-master-node-only">Editing the <code class="language-plaintext highlighter-rouge">conf/masters</code> (master node only)</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span><span class="nb">sudo </span>nano masters</code></pre></figure>
<p>It should look like</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span><span class="nb">cat </span>masters
192.168.103.24
hadoop@sys8:~/hadoop/conf<span class="err">$</span></code></pre></figure>
<h3 id="editing-the-confslaves-master-node-only">Editing the <code class="language-plaintext highlighter-rouge">conf/slaves</code> (master node only)</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span><span class="nb">sudo </span>nano slaves</code></pre></figure>
<p>It should look like</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span><span class="nb">cat </span>slaves
192.168.103.24
192.168.103.26
192.168.103.28
hadoop@sys8:~/hadoop/conf<span class="err">$</span></code></pre></figure>
<p>If you have additional slaves to add up, just add those in the <code class="language-plaintext highlighter-rouge">conf/slaves</code> file after a newline</p>
<hr />
<p>So we have basically added ip’s of all the nodes inside the <code class="language-plaintext highlighter-rouge">slaves</code> file.</p>
<p>###Generate the ssh keys for the master again</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span>ssh-keygen
Generating public/private rsa key pair.
Enter file <span class="k">in </span>which to save the key <span class="o">(</span>/home/hadoop/.ssh/id_rsa<span class="o">)</span>:
/home/hadoop/.ssh/id_rsa already exists.
Overwrite <span class="o">(</span>y/n<span class="o">)</span>? y
Enter passphrase <span class="o">(</span>empty <span class="k">for </span>no passphrase<span class="o">)</span>:
Enter same passphrase again:
Your identification has been saved <span class="k">in</span> /home/hadoop/.ssh/id_rsa.
Your public key has been saved <span class="k">in</span> /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
<span class="nb">fc</span>:73:90:c9:c9:cc:b7:13:e7:75:55:0b:94:b5:18:01 hadoop@sys8
The key<span class="s1">'s randomart image is:
+--[ RSA 2048]----+
| Eo=+..|
| .+ +|
| . o.|
| . = + .|
| S X o . o|
| . o = ..|
| o + . |
| o . |
| |
+-----------------+
hadoop@sys8:~/hadoop/conf$ </span></code></pre></figure>
<h3 id="add-this-key-to-all-the-nodes-including-the-master">Add this key to all the nodes (Including the master)</h3>
<p>For <code class="language-plaintext highlighter-rouge">sys9</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span>ssh-copy-id hadoop@sys9
The authenticity of host <span class="s1">'sys9 (192.168.103.26)'</span> can<span class="s1">'t be established.
ECDSA key fingerprint is e9:d3:5f:de:5b:0d:93:17:18:16:b8:9b:39:39:fa:62.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@sys9'</span>s password:
Number of key<span class="o">(</span>s<span class="o">)</span> added: 1
Now try logging into the machine, with: <span class="s2">"ssh 'hadoop@sys9'"</span>
and check to make sure that only the key<span class="o">(</span>s<span class="o">)</span> you wanted were added.
hadoop@sys8:~/hadoop/conf<span class="err">$</span></code></pre></figure>
<p>For <code class="language-plaintext highlighter-rouge">sys10</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span>ssh-copy-id hadoop@sys10
The authenticity of host <span class="s1">'sys10 (192.168.103.28)'</span> can<span class="s1">'t be established.
ECDSA key fingerprint is 1b:e2:6c:bc:2d:41:12:4d:79:e1:60:5c:08:74:32:9a.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@sys10'</span>s password:
Permission denied, please try again.
hadoop@sys10<span class="s1">'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '</span>hadoop@sys10<span class="s1">'"
and check to make sure that only the key(s) you wanted were added.
hadoop@sys8:~/hadoop/conf$</span></code></pre></figure>
<p>For <code class="language-plaintext highlighter-rouge">sys8</code> itself!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop/conf<span class="nv">$ </span>ssh-copy-id hadoop@sys8
The authenticity of host <span class="s1">'sys4 (192.168.103.24)'</span> can<span class="s1">'t be established.
ECDSA key fingerprint is 1b:e2:6c:bc:2d:41:12:4d:79:e1:60:5c:08:74:32:9a.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@sys4'</span>s password:
Permission denied, please try again.
hadoop@sys4<span class="s1">'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '</span>hadoop@sys4<span class="s1">'"
and check to make sure that only the key(s) you wanted were added.
hadoop@sys8:~/hadoop/conf$</span></code></pre></figure>
<h2 id="format-the-apptmp-directory-contents-master-node">Format the <code class="language-plaintext highlighter-rouge">/app/tmp/</code> directory contents (master node)</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo rm</span> <span class="nt">-rf</span> /app/hadoop/tmp/<span class="k">*</span>
hadoop@sys8:~<span class="err">$</span></code></pre></figure>
<h2 id="format-the-namenode-master-node">Format the namenode (master node)</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
</pre></td><td class="code"><pre>hadoop@sys8:~/hadoop<span class="nv">$ </span>bin/hadoop namenode <span class="nt">-format</span>
Warning: <span class="nv">$HADOOP_HOME</span> is deprecated.
15/09/13 23:11:22 INFO namenode.NameNode: STARTUP_MSG:
/<span class="k">************************************************************</span>
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host <span class="o">=</span> sys8/192.168.103.24
STARTUP_MSG: args <span class="o">=</span> <span class="o">[</span><span class="nt">-format</span><span class="o">]</span>
STARTUP_MSG: version <span class="o">=</span> 1.2.1
STARTUP_MSG: build <span class="o">=</span> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 <span class="nt">-r</span> 1503152<span class="p">;</span> compiled by <span class="s1">'mattf'</span> on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java <span class="o">=</span> 1.7.0_79
<span class="k">************************************************************</span>/
15/09/13 23:11:22 INFO util.GSet: Computing capacity <span class="k">for </span>map BlocksMap
15/09/13 23:11:22 INFO util.GSet: VM <span class="nb">type</span> <span class="o">=</span> 64-bit
15/09/13 23:11:22 INFO util.GSet: 2.0% max memory <span class="o">=</span> 932184064
15/09/13 23:11:22 INFO util.GSet: capacity <span class="o">=</span> 2^21 <span class="o">=</span> 2097152 entries
15/09/13 23:11:22 INFO util.GSet: <span class="nv">recommended</span><span class="o">=</span>2097152, <span class="nv">actual</span><span class="o">=</span>2097152
15/09/13 23:11:22 INFO namenode.FSNamesystem: <span class="nv">fsOwner</span><span class="o">=</span>hadoop
15/09/13 23:11:22 INFO namenode.FSNamesystem: <span class="nv">supergroup</span><span class="o">=</span>supergroup
15/09/13 23:11:22 INFO namenode.FSNamesystem: <span class="nv">isPermissionEnabled</span><span class="o">=</span><span class="nb">true
</span>15/09/13 23:11:22 INFO namenode.FSNamesystem: dfs.block.invalidate.limit<span class="o">=</span>100
15/09/13 23:11:22 INFO namenode.FSNamesystem: <span class="nv">isAccessTokenEnabled</span><span class="o">=</span><span class="nb">false </span><span class="nv">accessKeyUpdateInterval</span><span class="o">=</span>0 min<span class="o">(</span>s<span class="o">)</span>, <span class="nv">accessTokenLifetime</span><span class="o">=</span>0 min<span class="o">(</span>s<span class="o">)</span>
15/09/13 23:11:22 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length <span class="o">=</span> 0
15/09/13 23:11:22 INFO namenode.NameNode: Caching file names occuring more than 10 <span class="nb">times
</span>15/09/13 23:11:22 INFO common.Storage: Image file /app/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved <span class="k">in </span>0 seconds.
15/09/13 23:11:22 INFO namenode.FSEditLog: closing edit log: <span class="nv">position</span><span class="o">=</span>4, <span class="nv">editlog</span><span class="o">=</span>/app/hadoop/tmp/dfs/name/current/edits
15/09/13 23:11:22 INFO namenode.FSEditLog: close success: <span class="nb">truncate </span>to 4, <span class="nv">editlog</span><span class="o">=</span>/app/hadoop/tmp/dfs/name/current/edits
15/09/13 23:11:23 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
15/09/13 23:11:23 INFO namenode.NameNode: SHUTDOWN_MSG:
/<span class="k">************************************************************</span>
SHUTDOWN_MSG: Shutting down NameNode at sys8/192.168.103.24
<span class="k">************************************************************</span>/
hadoop@sys8:~/hadoop<span class="nv">$ </span>
</pre></td></tr></tbody></table></code></pre></figure>
<h2 id="start-the-name-node-in-master-node">Start the name node (in master node)</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
</pre></td><td class="code"><pre>hadoop@sys8:~/hadoop<span class="nv">$ </span>bin/start-all.sh
Warning: <span class="nv">$HADOOP_HOME</span> is deprecated.
starting namenode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-sys8.out
192.168.103.24: starting datanode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-sys8.out
192.168.103.28: starting datanode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-sys10.out
192.168.103.26: starting datanode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-sys9.out
192.168.103.24: starting secondarynamenode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-sys8.out
starting jobtracker, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-sys8.out
192.168.103.26: starting tasktracker, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-sys9.out
192.168.103.28: starting tasktracker, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-sys10.out
192.168.103.24: starting tasktracker, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-sys8.out
hadoop@sys8:~/hadoop<span class="err">$</span>
</pre></td></tr></tbody></table></code></pre></figure>
<h2 id="check-jps-in-all-systems">Check JPS (in all systems)</h2>
<p><code class="language-plaintext highlighter-rouge">sys8</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span>jps
3526 Jps
2817 NameNode
3258 JobTracker
3153 SecondaryNameNode
2976 DataNode
3442 TaskTracker
hadoop@sys8:~/hadoop<span class="err">$</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">sys9</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys9:~/hadoop<span class="nv">$ </span>jps
2440 TaskTracker
2519 Jps
hadoop@sys9:~/hadoop<span class="err">$</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">sys10</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys10:~/hadoop<span class="nv">$ </span>jps
2549 Jps
2469 TaskTracker
hadoop@sys10:~/hadoop<span class="err">$</span></code></pre></figure>
<h2 id="check-the-web-interface-in-your-browser">Check the web interface in your browser</h2>
<p>You can also check the web interface of hadoop in your browser</p>
<p>Using the url : <code class="language-plaintext highlighter-rouge">http://192.168.103.24:50030/</code></p>
<figure>
<a href="/images/machine_list_hadoop.jpg"><img src="/images/machine_list_hadoop.jpg" alt="" />
<figcaption>List of all nodes in the Hadoop Cluster</figcaption>
</figure>
***
<figure>
<a href="/images/machine_list_hadoop.jpg"><img src="/images/machine_list_hadoop.jpg" alt="" />
<figcaption>Mapreduce User Interface in the master node</figcaption>
</figure>
That's all Folks!
</a></figure></a></figure>
Install Hadoop(Single node)
2015-09-10T00:00:00+00:00
https://www.tasdikrahman.com/2015/09/10/Install-Hadoop-1.0.3-Single-Node-on-Ubuntu-14.04.2
<h2 id="on-a-starting-note-">On a starting note :</h2>
<p>I am assuming that you have a fresh Ubuntu install on your system as this will cut down a lot of frustration trying to debug why Hadoop is not running.</p>
<p>I am using <code class="language-plaintext highlighter-rouge">nano</code> as the text editor for this article but you can use any other text-editor like <code class="language-plaintext highlighter-rouge">vi</code>, <code class="language-plaintext highlighter-rouge">sublime</code> or <code class="language-plaintext highlighter-rouge">atom</code> for doing the same</p>
<blockquote>
<p>Next article : <a href="http://www.tasdikrahman.com/2015/09/13/Setup-Hadoop-on-Ubuntu-Multi-Node-Cluster/">Install Hadoop Multi Node 1.0.3 on Ubuntu 14.04.2</a></p>
</blockquote>
<h2 id="installation">Installation</h2>
<h3 id="create-a-dedicated-user">Create a dedicated user</h3>
<p>Create a seperate user named <code class="language-plaintext highlighter-rouge">hadoop</code> for seperating out the configuration files and the installation files</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sys8@sys8:~<span class="nv">$ </span><span class="nb">sudo </span>adduser hadoop</code></pre></figure>
<p>You would be then prompted to add the new UNIX password and details</p>
<p>add this user to the sudoer’s group</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sys8@sys8:~<span class="nv">$ </span><span class="nb">sudo </span>adduser hadoop <span class="nb">sudo</span></code></pre></figure>
<p>Switch to the newly created user</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sys8@sys8:~<span class="nv">$ </span>su - hadoop</code></pre></figure>
<p>Install JAVA which is in the default repos of Ubuntu</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>default-jdk</code></pre></figure>
<h3 id="configure-ssh">Configure SSH:</h3>
<p>Enabling ssh for localhost is the next step</p>
<p>Install <code class="language-plaintext highlighter-rouge">openssh-server</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>openssh-server</code></pre></figure>
<p>Generate the keys</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span>ssh-keygen <span class="nt">-t</span> rsa <span class="nt">-P</span> <span class="s2">""</span>
Generating public/private rsa key pair.
Enter file <span class="k">in </span>which to save the key <span class="o">(</span>/home/hadoop/.ssh/id_rsa<span class="o">)</span>:
Created directory <span class="s1">'/home/hadoop/.ssh'</span><span class="nb">.</span>
Your identification has been saved <span class="k">in</span> /home/hadoop/.ssh/id_rsa.
Your public key has been saved <span class="k">in</span> /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
68:b0:84:c0:3f:16:41:38:d9:7e:d6:63:a3:a0:28:f5 hadoop@sys8
The key<span class="s1">'s randomart image is:
+--[ RSA 2048]----+
|o =o. |
| * + |
| = + . |
| .B = * |
|..o.* = S |
|o. Eo |
|. |
| |
| |
+-----------------+
hadoop@sys8:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys</span></code></pre></figure>
<p>Copy the keys over to enable passwordless ssh</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span>ssh-copy-id <span class="nt">-i</span> <span class="nv">$HOME</span>/.ssh/id_rsa.pub hadoop@localhost
The authenticity of host <span class="s1">'localhost (127.0.0.1)'</span> can<span class="s1">'t be established.
ECDSA key fingerprint is 17:f4:fc:aa:88:4d:51:b1:08:ae:df:75:2f:07:37:26.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.</span></code></pre></figure>
<p>Test the ssh connection to the localhost</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span>ssh hadoop@localhost
Welcome to Ubuntu 14.04.3 LTS <span class="o">(</span>GNU/Linux 3.19.0-28-generic x86_64<span class="o">)</span>
<span class="k">*</span> Documentation: https://help.ubuntu.com/
60 packages can be updated.
24 updates are security updates.
hadoop@sys8:~<span class="nv">$ </span><span class="nb">exit
logout
</span>Connection to localhost closed.
hadoop@sys8:~<span class="err">$</span></code></pre></figure>
<h3 id="apache-hadoop-installation">Apache Hadoop Installation:</h3>
<p>Download hadoop from apache’s site,</p>
<p>But first change to hadoop’s home directory first</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">pwd</span>
/home/hadoop
hadoop@sys8:~<span class="nv">$ </span>wget https://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1.tar.gz
<span class="nt">--2015-09-12</span> 18:51:49-- https://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1.tar.gz
Resolving archive.apache.org <span class="o">(</span>archive.apache.org<span class="o">)</span>... 140.211.11.131, 192.87.106.229, 2001:610:1:80bc:192:87:106:229
Connecting to archive.apache.org <span class="o">(</span>archive.apache.org<span class="o">)</span>|140.211.11.131|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 63851630 <span class="o">(</span>61M<span class="o">)</span> <span class="o">[</span>application/x-gzip]
Saving to: ‘hadoop-1.2.1.tar.gz’
100%[<span class="o">======================================================================================================>]</span> 6,38,51,630 1.82MB/s <span class="k">in </span>42s
2015-09-12 18:52:32 <span class="o">(</span>1.45 MB/s<span class="o">)</span> - ‘hadoop-1.2.1.tar.gz’ saved <span class="o">[</span>63851630/63851630]</code></pre></figure>
<p>Extract and rename the file</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo tar </span>xzf hadoop-1.2.1.tar.gz
hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo mv </span>hadoop-1.2.1 hadoop</code></pre></figure>
<p>Set the user permissions</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo chown</span> <span class="nt">-R</span> hadoop:hadoop hadoop</code></pre></figure>
<hr />
<h2 id="configuration">Configuration</h2>
<h3 id="update-etcprofile">Update <code class="language-plaintext highlighter-rouge">/etc/profile</code></h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/profile</code></pre></figure>
<p>Add the following lines at the end</p>
<p><strong>NOTE</strong> :</p>
<p>This is assuming that you have installed the default-jdk from the repo’s. Change it accordingly for any other java version</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="code"><pre><span class="nb">export </span><span class="nv">HADOOP_HOME</span><span class="o">=</span><span class="s2">"/home/hadoop/hadoop/"</span>
<span class="nb">export </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="s2">"/usr/lib/jvm/java-1.7.0-openjdk-amd64"</span>
<span class="nb">unalias </span>fs &> /dev/null
<span class="nb">alias </span><span class="nv">fs</span><span class="o">=</span><span class="s2">"hadoop fs"</span>
<span class="nb">unalias </span>hls &> /dev/null
<span class="nb">alias </span><span class="nv">hls</span><span class="o">=</span><span class="s2">"fs -ls"</span>
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PATH</span><span class="s2">:</span><span class="nv">$HADOOP_HOME</span><span class="s2">/bin"</span>
</pre></td></tr></tbody></table></code></pre></figure>
<hr />
<h2 id="edit-the-configuration-files-for-hadoop">Edit the configuration files for Hadoop</h2>
<h3 id="edit-hadoop-envsh">Edit <code class="language-plaintext highlighter-rouge">hadoop-env.sh</code></h3>
<p>Change to the hadoop folder and add the JAVA home path</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~<span class="nv">$ </span><span class="nb">cd</span> ~/hadoop
hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">ls
</span>bin CHANGES.txt docs hadoop-core-1.2.1.jar hadoop-test-1.2.1.jar ivy.xml LICENSE.txt sbin webapps
build.xml conf hadoop-ant-1.2.1.jar hadoop-examples-1.2.1.jar hadoop-tools-1.2.1.jar lib NOTICE.txt share
c++ contrib hadoop-client-1.2.1.jar hadoop-minicluster-1.2.1.jar ivy libexec README.txt src
hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">sudo </span>nano conf/hadoop-env.sh</code></pre></figure>
<p>Add the following</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># The java implementation to use. Required.</span>
<span class="nb">export </span><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="s2">"/usr/lib/jvm/java-1.7.0-openjdk-amd64"</span>
<span class="nb">export </span><span class="nv">HADOOP_OPTS</span><span class="o">=</span><span class="nt">-Djava</span>.net.preferIPv4Stack<span class="o">=</span><span class="nb">true</span></code></pre></figure>
<p><strong>Note</strong> :</p>
<p>Added the second line so as to disable ipv6 specifically for hadoop and not for the whole system</p>
<p>###Create the <code class="language-plaintext highlighter-rouge">tmp</code> folder for Hadoop</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /app/hadoop/tmp
<span class="c">### add the permissions</span>
hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">sudo chown </span>hadoop:hadoop /app/hadoop/tmp
hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">sudo chmod </span>750 /app/hadoop/tmp</code></pre></figure>
<h3 id="edit-confcore-sitexml">Edit <code class="language-plaintext highlighter-rouge">conf/core-site.xml</code></h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">sudo </span>nano conf/core-site.xml</code></pre></figure>
<p>You would be having blank spaces in between the <configuration> tags</configuration></p>
<p>It should look something like this after editing</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
</pre></td><td class="code"><pre><span class="cp"><?xml version="1.0"?></span>
<span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span>
<span class="c"><!-- Put site-specific property overrides in this file. --></span>
<span class="nt"><configuration></span>
<span class="nt"><property></span>
<span class="nt"><name></span>hadoop.tmp.dir<span class="nt"></name></span>
<span class="nt"><value></span>/app/hadoop/tmp<span class="nt"></value></span>
<span class="nt"><description></span>A base for other temporary directories.<span class="nt"></description></span>
<span class="nt"></property></span>
<span class="nt"><property></span>
<span class="nt"><name></span>fs.default.name<span class="nt"></name></span>
<span class="nt"><value></span>hdfs://localhost:54310<span class="nt"></value></span>
<span class="nt"><description></span>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.<span class="nt"></description></span>
<span class="nt"></property></span>
<span class="nt"></configuration></span>
</pre></td></tr></tbody></table></code></pre></figure>
<h3 id="edit-confmapred-sitexml">Edit <code class="language-plaintext highlighter-rouge">conf/mapred-site.xml</code></h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">sudo </span>nano conf/mapred-site.xml</code></pre></figure>
<p>It should look something like this after editing</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="code"><pre><span class="cp"><?xml version="1.0"?></span>
<span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span>
<span class="c"><!-- Put site-specific property overrides in this file. --></span>
<span class="nt"><configuration></span>
<span class="nt"><property></span>
<span class="nt"><name></span>mapred.job.tracker<span class="nt"></name></span>
<span class="nt"><value></span>localhost:54311<span class="nt"></value></span>
<span class="nt"><description></span>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
<span class="nt"></description></span>
<span class="nt"></property></span>
<span class="nt"></configuration></span>
</pre></td></tr></tbody></table></code></pre></figure>
<h3 id="edit-confhdfs-sitexml">Edit <code class="language-plaintext highlighter-rouge">conf/hdfs-site.xml</code></h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span><span class="nb">sudo </span>nano conf/hdfs-site.xml</code></pre></figure>
<p>It should look something like this after editing</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="code"><pre><span class="cp"><?xml version="1.0"?></span>
<span class="cp"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span>
<span class="c"><!-- Put site-specific property overrides in this file. --></span>
<span class="nt"><configuration></span>
<span class="nt"><property></span>
<span class="nt"><name></span>dfs.replication<span class="nt"></name></span>
<span class="nt"><value></span>1<span class="nt"></value></span>
<span class="nt"><description></span>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
<span class="nt"></description></span>
<span class="nt"></property></span>
<span class="nt"></configuration></span>
</pre></td></tr></tbody></table></code></pre></figure>
<h3 id="format-the-hdfs-file">Format the HDFS file</h3>
<p><strong>Note</strong> : Run this one command only once, i.e now</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span>bin/hadoop namenode <span class="nt">-format</span>
15/09/12 19:01:48 INFO namenode.NameNode: STARTUP_MSG:
/<span class="k">************************************************************</span>
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host <span class="o">=</span> sys8/127.0.1.1
STARTUP_MSG: args <span class="o">=</span> <span class="o">[</span><span class="nt">-format</span><span class="o">]</span>
STARTUP_MSG: version <span class="o">=</span> 1.2.1
STARTUP_MSG: build <span class="o">=</span> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 <span class="nt">-r</span> 1503152<span class="p">;</span> compiled by <span class="s1">'mattf'</span> on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java <span class="o">=</span> 1.7.0_79
<span class="k">************************************************************</span>/
15/09/12 19:01:49 INFO util.GSet: Computing capacity <span class="k">for </span>map BlocksMap
15/09/12 19:01:49 INFO util.GSet: VM <span class="nb">type</span> <span class="o">=</span> 64-bit
15/09/12 19:01:49 INFO util.GSet: 2.0% max memory <span class="o">=</span> 932184064
15/09/12 19:01:49 INFO util.GSet: capacity <span class="o">=</span> 2^21 <span class="o">=</span> 2097152 entries
15/09/12 19:01:49 INFO util.GSet: <span class="nv">recommended</span><span class="o">=</span>2097152, <span class="nv">actual</span><span class="o">=</span>2097152
15/09/12 19:01:49 INFO namenode.FSNamesystem: <span class="nv">fsOwner</span><span class="o">=</span>hadoop
15/09/12 19:01:49 INFO namenode.FSNamesystem: <span class="nv">supergroup</span><span class="o">=</span>supergroup
15/09/12 19:01:49 INFO namenode.FSNamesystem: <span class="nv">isPermissionEnabled</span><span class="o">=</span><span class="nb">true
</span>15/09/12 19:01:49 INFO namenode.FSNamesystem: dfs.block.invalidate.limit<span class="o">=</span>100
15/09/12 19:01:49 INFO namenode.FSNamesystem: <span class="nv">isAccessTokenEnabled</span><span class="o">=</span><span class="nb">false </span><span class="nv">accessKeyUpdateInterval</span><span class="o">=</span>0 min<span class="o">(</span>s<span class="o">)</span>, <span class="nv">accessTokenLifetime</span><span class="o">=</span>0 min<span class="o">(</span>s<span class="o">)</span>
15/09/12 19:01:49 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length <span class="o">=</span> 0
15/09/12 19:01:49 INFO namenode.NameNode: Caching file names occuring more than 10 <span class="nb">times
</span>15/09/12 19:01:49 INFO common.Storage: Image file /app/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved <span class="k">in </span>0 seconds.
15/09/12 19:01:49 INFO namenode.FSEditLog: closing edit log: <span class="nv">position</span><span class="o">=</span>4, <span class="nv">editlog</span><span class="o">=</span>/app/hadoop/tmp/dfs/name/current/edits
15/09/12 19:01:49 INFO namenode.FSEditLog: close success: <span class="nb">truncate </span>to 4, <span class="nv">editlog</span><span class="o">=</span>/app/hadoop/tmp/dfs/name/current/edits
15/09/12 19:01:49 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
15/09/12 19:01:49 INFO namenode.NameNode: SHUTDOWN_MSG:
/<span class="k">************************************************************</span>
SHUTDOWN_MSG: Shutting down NameNode at sys8/127.0.1.1
<span class="k">************************************************************</span>/</code></pre></figure>
<hr />
<p>##Start the Single node</p>
<p><strong>NOTE</strong> :</p>
<p><code class="language-plaintext highlighter-rouge">bin/start-all.sh</code> is deprecated so we will use <code class="language-plaintext highlighter-rouge">bin/start-dfs.sh</code> and then <code class="language-plaintext highlighter-rouge">bin/start-mapred.dfs</code></p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span>bin/start-dfs.sh
starting namenode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-namenode-sys9.out
localhost: starting datanode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-datanode-sys9.out
localhost: starting secondarynamenode, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-sys9.out
hadoop@sys8:~/hadoop<span class="nv">$ </span>bin/start-mapred.sh
starting jobtracker, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-sys9.out
localhost: starting tasktracker, logging to /home/hadoop/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-sys9.out</code></pre></figure>
<h3 id="check-if-everything-is-running-fine">Check if everything is running fine:</h3>
<p>Run <code class="language-plaintext highlighter-rouge">jps</code> for that and the output should look something like this</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span>jps
11508 TaskTracker
10938 NameNode
11868 Jps
11250 SecondaryNameNode
11347 JobTracker
11085 DataNode
hadoop@sys8:~/hadoop<span class="nv">$ </span></code></pre></figure>
<hr />
<h2 id="stopping-hadoop">Stopping hadoop</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">hadoop@sys8:~/hadoop<span class="nv">$ </span>bin/stop-all.sh</code></pre></figure>
<p>That’s all Folks!</p>
ROS Jade : Installation
2015-08-28T00:00:00+00:00
https://www.tasdikrahman.com/2015/08/28/Install-ROS-Jade-on-Ubuntu-14.10
<h3 id="what-the-heck-is-ros-anyway-">What the heck is ROS anyway :</h3>
<p>Robot Operating System (ROS) is a collection of software frameworks for robot software development, providing operating system-like functionality on a heterogeneous computer cluster.</p>
<p>So in layman terms, it just helps us build robot applications.</p>
<blockquote>
<p>Previous post on ROS : <a href="http://www.tasdikrahman.com/2015/08/28/Configure-ROS-Jade-on-Ubuntu-14.10/">Configuring ROS - Jade on Ubuntu 14.10</a></p>
</blockquote>
<h3 id="note-">Note :</h3>
<p>ROS Jade ONLY supports Trusty (14.04), Utopic (14.10) and Vivid (15.04) for debian packages.
If you are on any other version of Ubuntu or have a different flavor of linux installed, I suggest you head over
to <a href="http://wiki.ros.org/">wiki.ros.org</a>.</p>
<p>Note that this guide is written keeping in mind that we are on Ubuntu 14.10 !</p>
<h3 id="requirements-">Requirements :</h3>
<ul>
<li>Supported OS :
<ul>
<li>Ubuntu Trusty(14.04)</li>
<li>Ubuntu Utopic (14.10)</li>
<li>Ubuntu Vivid (15.04)</li>
</ul>
</li>
<li>Minimum requirements :
<ul>
<li>C++03</li>
<li>C++11 features are not used, but code should compile when -std=c++11 is used</li>
<li>Python 2.7
*Python 3.3 not required, but testing against it is recommended</li>
<li>Lisp SBCL 1.1.14</li>
<li>CMake 2.8.12</li>
<li>Boost 1.54</li>
</ul>
</li>
</ul>
<h3 id="installation-">Installation :</h3>
<ol>
<li><strong>Configure your Ubuntu repositories</strong> :
<ul>
<li>Configure your Ubuntu repositories to allow “restricted,” “universe,” and “multiverse.” You can
follow <a href="https://help.ubuntu.com/community/Repositories/Ubuntu">follow the Ubuntu guide </a> for completing this work.</li>
</ul>
</li>
<li>
<p><strong>Set up your sources.list</strong></p>
<p>This can be done by running the following command in the terminal</p>
</li>
</ol>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>sh <span class="nt">-c</span> <span class="s1">'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'</span>
</code></pre></figure>
<ol>
<li><strong>Set up your keys</strong></li>
</ol>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-key adv <span class="nt">--keyserver</span> hkp://pool.sks-keyservers.net <span class="nt">--recv-key</span> 0xB01FA116
</code></pre></figure>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Note that if you are behind a proxy, this command wont work. You will have to workaround that I guess(use a dongle maybe!)
</code></pre></div></div>
<ol>
<li><strong>Installation</strong></li>
</ol>
<p>Make sure that Debian package index is up-to-date.</p>
<p>To make sure just do a</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get update
</code></pre></figure>
<p>and you are good to go</p>
<h4 id="for-those-running-14042">For those running 14.04.2</h4>
<p>If you are on 14.04.2, you have to manually fix the dependy issues,</p>
<blockquote>
<p>DO NOT INSTALL THE BELOW PACKAGES IF YOU ARE ON 14.04, THIS WILL DESTROY YOUR X-SERVER. IN SHORT, YOU WONT BE ABLE TO SEE THE GUI OF YOUR OS AGAIN!</p>
</blockquote>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>xserver-xorg-dev-lts-utopic mesa-common-dev-lts-utopic libxatracker-dev-lts-utopic libopenvg1-mesa-dev-lts-utopic libgles2-mesa-dev-lts-utopic libgles1-mesa-dev-lts-utopic libgl1-mesa-dev-lts-utopic libgbm-dev-lts-utopic libegl1-mesa-dev-lts-utopic
</code></pre></figure>
<blockquote>
<p>DO NOT INSTALL THE ABOVE PACKAGES IF YOU ARE ON 14.04, THIS WILL DESTROY YOUR X-SERVER. IN SHORT, YOU WONT BE ABLE TO SEE THE GUI OF YOUR OS AGAIN!</p>
</blockquote>
<p>Alternatively you can try,</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>libgl1-mesa-dev-lts-utopic
</code></pre></figure>
<p>to fix the dependency issues.</p>
<h4 id="for-those-running-1404-1410-and-1504">For those running 14.04, 14.10 and 15.04</h4>
<p>After that you can run the command to install the <strong>Desktop-Full Install</strong> which is the <strong>recommeded</strong> one by
doing.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"> tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>ros-jade-desktop-full
</code></pre></figure>
<p>If everything is good uptil here, you should see the package manager downloading the required packages.</p>
<h3 id="get-yourself-a-coffee-or-two"><strong>Get yourself a coffee or two</strong></h3>
<p>because it can take a helluva a time depending on the speed of your internet connection.</p>
<h3 id="initialize-rosdep">Initialize rosdep</h3>
<p>Before you can use ROS, you will need to initialize rosdep. rosdep enables you to easily install system
dependencies for source you want to compile and is required to run some core components in ROS.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>rosdep init
tasdik@Acer:~<span class="nv">$ </span>rosdep update</code></pre></figure>
<h3 id="environment-setup">Environment setup</h3>
<p>It’s convenient if the ROS environment variables are automatically added to your bash session every time
a new shell is launched:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">echo</span> <span class="s2">"source /opt/ros/jade/setup.bash"</span> <span class="o">>></span> ~/.bashrc
tasdik@Acer:~<span class="nv">$ </span><span class="nb">source</span> ~/.bashrc</code></pre></figure>
<h3 id="getting-rosinstall">Getting rosinstall</h3>
<p><a href="http://wiki.ros.org/rosinstall">rosinstall</a> is a frequently used command-line tool in ROS that is
distributed separately. It enables you to easily download many source trees for ROS packages with one command.</p>
<p>For ubuntu users, just run</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>python-rosinstall</code></pre></figure>
<h3 id="finally">Finally</h3>
<p>The next step would be to <a href="http://www.tasdikrahman.com/Configure-ROS-Jade-on-Ubuntu-14.10/">configure ROS, the steps of which can be found here</a></p>
<p>Now that you have installed ROS on your system, you can look forward to the
<a href="http://wiki.ros.org/ROS/Tutorials">ROS tutorials</a></p>
<p>Till then Goodbye!</p>
<h3 id="references-">References :</h3>
<p>I could not have written this guide if the documentation had not been so crisp and straight forward.
Kudos to the ROS development team!</p>
<ul>
<li><a href="http://wiki.ros.org/jade/Installation/Ubuntu">http://wiki.ros.org/jade/Installation/Ubuntu</a></li>
</ul>
ROS Jade : Configuration
2015-08-28T00:00:00+00:00
https://www.tasdikrahman.com/2015/08/28/Configure-ROS-Jade-on-Ubuntu-14.10
<h3 id="configuring-ros">Configuring ROS</h3>
<p>If you have not installed ROS, I have written a short guide describing the process. It can be found here found here</p>
<blockquote>
<p><a href="https://www.tasdikrahman.com/2015/08/28/Install-ROS-Jade-on-Ubuntu-14.10/">Install ROS - Jade on Ubuntu 14.10</a></p>
</blockquote>
<p>The first step is to check whether the environment variables for ROS are setup properly
Do a</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">export</span> | <span class="nb">grep </span>ROS</code></pre></figure>
<p>Look for <code class="language-plaintext highlighter-rouge">ROS_ROOT</code> and <code class="language-plaintext highlighter-rouge">ROS_PACKAGE_PATH</code> whether they are set.</p>
<p>It should look something like this</p>
<figure>
<a href="/images/path_ros.jpg"><img src="/images/path_ros.jpg" alt="" />
<figcaption>List of all nodes in the Hadoop Cluster</figcaption>
</figure>
If not, just do a
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">source</span> /opt/ros/<distro>/setup.bash</code></pre></figure>
where <distro> is <del>the distribution name of you OS.</del> version of ROS installed (Jade Turtle for our case).
### Creating ROS Workspace
I will be using [catkin](http://wiki.ros.org/catkin) to create my workspace.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">mkdir</span> <span class="nt">-p</span> ~/catkin_ws/src
tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd</span> ~/catkin_ws/src
tasdik@Acer:~<span class="nv">$ </span>catkin_init_workspace</code></pre></figure>
Even though the folder is empty(we just have a file named CMakeLists.txt). We can still build the workspace.
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span>catkin_make
tasdik@Acer:~<span class="nv">$ </span><span class="nb">cd</span> ~/catkin_ws/</code></pre></figure>
Do an `ls` and now you can see that we have folders like `build`, `devel` folder.
Inside the `devel` folder, there are several `*.sh` files
## Source the setup file
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/catkin_ws<span class="nv">$ </span><span class="nb">source </span>devel/setup.bash</code></pre></figure>
To make sure, everything is done correctly so far, do
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~/catkin_ws<span class="nv">$ </span><span class="nb">echo</span> <span class="nv">$ROS_PACKAGE_PATH</span>
/opt/ros/jade/share:/opt/ros/jade/stacks
tasdik@Acer:~/catkin_ws<span class="err">$</span></code></pre></figure>
If that is your Output for the prompt, you are good to go.
Till then Goodbye!
### References :
* [http://wiki.ros.org/ROS/Tutorials/InstallingandConfiguringROSEnvironment](http://wiki.ros.org/ROS/Tutorials/InstallingandConfiguringROSEnvironment)
</distro></a></figure>
Apache2 : Virtual Hosts
2015-08-21T00:00:00+00:00
https://www.tasdikrahman.com/2015/08/21/Setup-Website-on-localhost-using-LAMP-Stack
<h2 id="what-do-we-need">What do we need?</h2>
<p>The two most common tools for this are the Apache and nginx servers.</p>
<h3 id="notes">Notes:</h3>
<p>You’ll need to edit a few system configuration files. If you’re uncomfortable with vim, replace vim with nano, or gedit in the following commands. For example, sudo vim will become <code class="language-plaintext highlighter-rouge">sudo -H gedit</code> or <code class="language-plaintext highlighter-rouge">sudo nano</code>.</p>
<p>Once you’re done setting it up, have a look at How to avoid using sudo when working in <code class="language-plaintext highlighter-rouge">/var/www</code>?
A more detailed guide is available from the Ubuntu LTS Server Guide.</p>
<h3 id="first-install-apache">First, install Apache:</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>apache2</code></pre></figure>
<p>The Apache configuration files are located in <code class="language-plaintext highlighter-rouge">/etc/apache2</code>. You’ll typically be interested in:</p>
<ul>
<li>
<p><code class="language-plaintext highlighter-rouge">/etc/apache2/sites-available</code> - contains the Virtual Host definitions. Definitions are enabled and disabled using the <code class="language-plaintext highlighter-rouge">a2ensite</code> and <code class="language-plaintext highlighter-rouge">a2dissite</code>commands. The enabled site definitions are linked to <code class="language-plaintext highlighter-rouge">/etc/apache2/sites-enabled</code>.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">/etc/apache2/conf-available</code> - contains custom configuration files. They are enabled and disabled using the <code class="language-plaintext highlighter-rouge">a2enconf</code> and <code class="language-plaintext highlighter-rouge">a2disconf</code> commands.
The enabled site configuration files are linked to <code class="language-plaintext highlighter-rouge">/etc/apache2/conf-enabled</code>.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">/var/www/html</code> - the default directory that Apache serves.</p>
</li>
<li>
<p>For most instructions, I’ll assume we are in <code class="language-plaintext highlighter-rouge">/etc/apache2</code>.</p>
</li>
</ul>
<h3 id="virtualhost-setup">VirtualHost setup</h3>
<p>Let us create a new site. There’s a default configuration available in <code class="language-plaintext highlighter-rouge">sites-enabled/default.conf</code>. We will make a copy of this, and work on it:</p>
<p>This is where should be</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:/etc/apache2<span class="nv">$ </span><span class="nb">ls
</span>apache2.conf conf-available conf-enabled envvars magic mods-available mods-enabled ports.conf sites-available sites-enabled
tasdik@Acer:/etc/apache2<span class="nv">$ </span></code></pre></figure>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo cp </span>sites-available/000-default.conf sites-available/my-name.conf
tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>nano sites-available/my-name.conf</code></pre></figure>
<p>It should look something like this</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
</pre></td><td class="code"><pre><VirtualHost <span class="k">*</span>:80>
<span class="c"># The ServerName directive sets the request scheme, hostname and port that</span>
<span class="c"># the server uses to identify itself. This is used when creating</span>
<span class="c"># redirection URLs. In the context of virtual hosts, the ServerName</span>
<span class="c"># specifies what hostname must appear in the request's Host: header to</span>
<span class="c"># match this virtual host. For the default virtual host (this file) this</span>
<span class="c"># value is not decisive as it is used as a last resort host regardless.</span>
<span class="c"># However, you must set it for any further virtual host explicitly.</span>
ServerName myname.com
ServerAdmin webmaster@localhost
DocumentRoot /var/www/my-name
<span class="c"># Available loglevels: trace8, ..., trace1, debug, info, notice, warn,</span>
<span class="c"># error, crit, alert, emerg.</span>
<span class="c"># It is also possible to configure the loglevel for particular</span>
<span class="c"># modules, e.g.</span>
<span class="c">#LogLevel info ssl:warn</span>
ErrorLog <span class="k">${</span><span class="nv">APACHE_LOG_DIR</span><span class="k">}</span>/error.log
CustomLog <span class="k">${</span><span class="nv">APACHE_LOG_DIR</span><span class="k">}</span>/access.log combined
<span class="c"># For most configuration files from conf-available/, which are</span>
<span class="c"># enabled or disabled at a global level, it is possible to</span>
<span class="c"># include a line for only one particular virtual host. For example the</span>
<span class="c"># following line enables the CGI configuration for this host only</span>
<span class="c"># after it has been globally disabled with "a2disconf".</span>
<span class="c">#Include conf-available/serve-cgi-bin.conf</span>
</VirtualHost>
<span class="c"># vim: syntax=apache ts=4 sw=4 sts=4 sr noet</span>
</pre></td></tr></tbody></table></code></pre></figure>
<h3 id="save-the-file-and-enable-it">Save the file, and enable it:</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>a2ensite my-name</code></pre></figure>
<p>Now, we need to set up the directory for the site:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo mkdir</span> /var/www/my-name</code></pre></figure>
<p>We’ll set permissions for convenience:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo chown</span> <span class="nv">$USER</span>:www-data /var/www/my-name
tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo chmod </span>g+s /var/www/my-name</code></pre></figure>
<p>Add a few HTML files here.</p>
<p>Since the virtual host is to run locally, we need to map myname.com to a local address. To do this, we need to edit <code class="language-plaintext highlighter-rouge">/etc/hosts</code>:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/hosts</code></pre></figure>
<p>It should look something like this</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><table class="rouge-table"><tbody><tr><td class="gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="code"><pre>127.0.0.1 localhost
127.0.1.1 Acer
127.0.0.2 myname.com myname
<span class="c"># The following lines are desirable for IPv6 capable hosts</span>
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
</pre></td></tr></tbody></table></code></pre></figure>
<h3 id="save-and-then-restart-apache">Save, and then restart Apache:</h3>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">tasdik@Acer:~<span class="nv">$ </span><span class="nb">sudo </span>service apache2 restart</code></pre></figure>
<p>Now, you can browse to <code class="language-plaintext highlighter-rouge">http://myname.com</code> or <code class="language-plaintext highlighter-rouge">http://myname</code>, and the contents of <code class="language-plaintext highlighter-rouge">/var/www/my-name</code> will be displayed.</p>