tag:blogger.com,1999:blog-55733556758065795112024-03-05T07:15:32.928-06:00Climb That MountainAttila Domokos' blogAttila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comBlogger79125tag:blogger.com,1999:blog-5573355675806579511.post-66319726689120453212018-11-14T13:57:00.002-06:002018-11-14T13:57:56.354-06:00Hspec with QuickCheck<p>This blog post will not describe what property testing is, there are other writings about it. Instead, I'll try to show you how to get started with it, as finding introductory tutorials on the topic is scarce, this is my attempt to fill that void.</p>
<p>I heard about property-based testing in 2015, when I attended Michael Nygaard’s StrangeLoop <a href="https://www.youtube.com/watch?v=N5HyVUPuU0E">talk</a>. As a long time TDD-er, I was highly skeptical of generating tests for my own code, but the idea started to grow on me.</p>
<p>Where would I use property testing in a real project? I was presented with a task at my job to solve this: "Write QuickCheck tests for a loan amortization algorithm. It has a starting amount, a term length (in month) and an interest rate. The test should verify:</p>
<ul>
<li>The loan is paid back</li>
<li>The term length matches the input argument</li>
<li>The principal is shrinking every month"</li>
</ul>
<p>I was eager to work on this task, however, practical examples were limited. The best information was in the book <a href="http://haskellbook.com/">Haskell Programming from First Principles</a>. I kept coming back to its chapter on testing to pick up what I know.</p>
<p>I wanted to learn with a simple example, I chose the Roman Numeral kata. The converter has 2 functions:</p>
<ul>
<li>Convert Arabic to Roman (<code>convertToRoman</code>)</li>
<li>Convert the Roman back to Arabic (<code>convertFromRoman</code>)</li>
</ul>
<p>Following the <a href="https://www.schoolofhaskell.com/user/pbv/an-introduction-to-quickcheck-testing">reverse list example</a> should be trivial: converting an Arabic number to Roman and back to Arabic should give the same number. The Haskell Book is using a similar example with morse code conversions.</p>
<p>I'll go through the good old TDD solution of the Roman Numeral kata in less detail, as the goal of this post is to describe how to use QuickCheck.</p>
<p>Here is how the code starts out:</p>
<pre class="brush: haskell">
module RomanNumeralsSpec where
import Test.Hspec
main :: IO ()
main = hspec spec
type Roman = String
convertToRoman :: Int -> Roman
convertToRoman 1 = "I"
convertToRoman 2 = "II"
convertToRoman 3 = "III"
convertToRoman 4 = "IV"
convertFromRoman :: Roman -> Int
convertFromRoman "" = undefined
spec :: Spec
spec =
describe "Converting to Roman Numerals" $ do
it "converts 1 to I" $
convertToRoman 1 `shouldBe` "I"
it "converts 2 to II" $
convertToRoman 2 `shouldBe` "II"
it "converts 3 to III" $
convertToRoman 3 `shouldBe` "III"
it "converts 4 to IV" $
convertToRoman 4 `shouldBe` "IV"
</pre>
<p>Look how dumb the test cases (and the code itself) are so far. When I introduce custom data types and a rule table for the conversions, the code becomes much simpler. Watch this:</p>
<pre class="brush: haskell">
...
type Conversions = [(Int, Roman)]
conversions :: Conversions
conversions =
[ (4, "IV")
, (1, "I") ]
convertToRoman :: Int -> Roman
convertToRoman 0 = []
convertToRoman x =
roman ++ convertToRoman (x - number)
where
(number, roman) =
head . filter (\(a,_) -> a <= x) $ conversions
...
</pre>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/76693b3da61eae126a6c70e76630ae54e266100e">Commit Point</a></p>
<p>Extending this now is easy, I don't have to modify the logic, just add more values to the conversions table.</p>
<pre class="brush: haskell">
...
conversions :: Conversions
conversions =
[ (90, "XC")
, (50, "L")
, (40, "XL")
, (10, "X")
, (9, "IX")
, (5, "V")
, (4, "IV")
, (1, "I") ]
...
spec :: Spec
spec =
describe "Converting to Roman Numerals" $ do
it "converts 1 to I" $
convertToRoman 1 `shouldBe` "I"
it "converts 2 to II" $
convertToRoman 2 `shouldBe` "II"
it "converts 3 to III" $
convertToRoman 3 `shouldBe` "III"
it "converts 4 to IV" $
convertToRoman 4 `shouldBe` "IV"
it "converts 5 to V" $
convertToRoman 5 `shouldBe` "V"
it "converts 6 to VI" $
convertToRoman 6 `shouldBe` "VI"
it "converts 8 to VIII" $
convertToRoman 8 `shouldBe` "VIII"
it "converts 9 to IX" $
convertToRoman 9 `shouldBe` "IX"
it "converts 10 to X" $
convertToRoman 10 `shouldBe` "X"
it "converts 11 to XI" $
convertToRoman 11 `shouldBe` "XI"
it "converts 99 to L" $
convertToRoman 99 `shouldBe` "XCIX"
</pre>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/11ad84525034d586bf3e6127078bf70a6b26b9ef">Commit point</a></p>
<p>Oh, these tests are ugly. I'll make them a bit worse before I improve them, sorry about that.</p>
<p>The conversions table can be used to convert Roman numbers to Arabic. The Arabic value is accumulated by adding the matched values together.</p>
<p>Here are the conversion code and the accompanying tests:</p>
<pre class="brush: haskell">
...
import Data.List (isPrefixOf, find)
import Data.Maybe (fromJust)
...
convertFromRoman :: Roman -> Int
convertFromRoman "" = 0
convertFromRoman r =
number + convertFromRoman (drop (length roman) r)
where
(number, roman) = fromJust $ find (\(_,r') -> r' `isPrefixOf` r) conversions
...
describe "Roman to Number Conversions" $ do
it "converts I to 1" $
convertFromRoman "I" `shouldBe` 1
it "converts II to 2" $
convertFromRoman "II" `shouldBe` 2
it "converts III to 3" $
convertFromRoman "III" `shouldBe` 3
it "converts IV to 4" $
convertFromRoman "IV" `shouldBe` 4
it "converts V to 5" $
convertFromRoman "V" `shouldBe` 5
it "converts VIII to 8" $
convertFromRoman "VIII" `shouldBe` 8
it "converts IX to 9" $
convertFromRoman "IX" `shouldBe` 9
it "converts X to 10" $
convertFromRoman "X" `shouldBe` 10
it "converts XI to 11" $
convertFromRoman "XI" `shouldBe` 11
it "converts XCIX to 99" $
convertFromRoman "XCIX" `shouldBe` 99
</pre>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/e23149db832b3e1d83115e42063febd2f903ae35">Commit Point</a></p>
<p>I ended up with 47 lines of mindless, verbose and repeated test code.</p>
<p>Let's fire up GHCi, and see how this works in the REPL. Use the Makefile target <a href="https://github.com/adomokos/roman-numerals-hs/blob/master/Makefile#L17-L19">make repl-test</a> to try this:</p>
<pre class="brush: shell">
λ> convertToRoman 12
"XII"
λ> convertFromRoman . convertToRoman $ 12
12
</code></pre>
<p>OK, I can convert an Arabic number to Roman and convert that back to Arabic, I'll use this mechanism to verify the logic with QuickCheck.</p>
<p>Let's explore QuickCheck in the REPL. Please follow along, but notice, that the generated random numbers are going to be different for you.</p>
<pre class="brush: shell">
λ> import Test.QuickCheck
λ> sample (arbitrary :: Gen Int)
0
-1
-3
-4
-2
-9
8
14
11
-18
-16
</pre>
<p>It works for Strings as well:</p>
<pre class="brush: shell">
λ> sample (arbitrary :: Gen String)
""
"Q\EOT"
"#8"
"^x"
"&}\t\NULU k"
"\DC4\816800<"
"~!oH\763194&e\GSG"
"\STX\869030\194889\1040760\820031\799098w\SIHz"
"\47452\ETX>j\686979\ACK?\1094610\160069\943268\99807+\519462"
""
"\694257s\883691\996507fO_n4m\tF\357900"
</pre>
<p>Ok, this is great, but what if you just want random numbers between 1 and 3? Use the <code>elements</code> function:</p>
<pre class="brush: shell">
λ> let oneThroughThree = elements [1..3] :: Gen Int
oneThroughThree :: Gen Int
λ> sample' oneThroughThree
[3,1,3,3,3,1,2,1,3,2,2]
</pre>
<p>Note that I used the <code>sample'</code> function, which returns a list of elements now.</p>
<p>In case you want to have three with more frequencies, just provide a list accordingly. Like this:</p>
<pre class="brush: shell">
λ> let moreThrees = elements [1,2,3,3,3,3] :: Gen Int
moreThrees :: Gen Int
λ> sample' moreThrees
[1,3,3,3,3,2,3,2,1,1,3]
</pre>
<p>In the sample list the 3s are represented with higher frequencies.</p>
<p>But what if you want a combination of Strings and Integers for your needs? Well, you can build a short function for that:</p>
<pre class="brush: haskell">
import Test.QuickCheck
...
genTuple :: (Arbitrary a, Arbitrary b)
=> Gen (a, b)
genTuple = do
a <- arbitrary
b <- arbitrary
return (a, b)
</pre>
<p>Fire up the REPL with the test code (I used the included Makefile's <code>repl-test</code> target), and sample this new function to get a list of tuples with Integers and Strings:</p>
<pre class="brush: shell">
λ> import Test.QuickCheck
λ> sample' (genTuple :: Gen (Int, String))
[(0,""),(-1,"P!"),(0,"\703651\888426O"),(-5,"\DLEs_\640436>\a"),(-6,""),(10,"T\45432\&5?\STX.j"),(-10,"h\266318\SUB\175378"),(9,"\ESC\978066"),(16,"FQ@w;'I^\EM\NUL"),(7,""),(12,"\ACK\569630\49462")]
</pre>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/c25a9e4f45bd78f067d87c8c9f19f389ea3104da">Commit Point</a></p>
<p>After this brief intro, let's replace the manual tests with generated tests by QuickCheck.</p>
<p>This is the function I came up with to test the logic:</p>
<pre class="brush: haskell">
prop_convertNumber :: Int -> Bool
prop_convertNumber x = (convertFromRoman . convertToRoman) x == x
</pre>
<p>I gave it a try in the REPL:</p>
<pre class="brush: shell">
λ> import Test.QuickCheck
λ> quickCheck prop_convertNumber
*** Failed! Exception: 'Prelude.head: empty list' (after 3 tests and 1 shrink):
-1
λ> quickCheck prop_convertNumber
*** Failed! Exception: 'Prelude.head: empty list' (after 6 tests and 1 shrink):
-1
</pre>
<p>Oh, crap… QuickCheck generated <code>-1</code>, but my logic only works with positive numbers. I have to tell QuickCheck what numbers it can use.</p>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/d872da41f55c120d58d18d686fce90578d3490af">Commit Point</a></p>
<p>A valid range needs to be passed to QuickCheck to exercise the number conversion logic, the <code>elements</code> function could do just that. I add these two functions to my test:</p>
<pre class="brush: haskell">
numbers :: [Int]
numbers = [1..1000]
genNumbers :: Gen Int
genNumbers = elements numbers
</pre>
<p>I could easily verify in the REPL that the <code>genNumbers</code> function is producing numbers within the defined range:</p>
<pre class="brush: shell">
λ> import Test.QuickCheck
λ> sample' genNumbers
[745,853,321,678,436,711,825,593,441,900,315]
λ> sample' genNumbers
[706,110,263,36,807,589,555,444,60,261,116]
</pre>
<p>Great, only positive numbers are generated, I can use this set for my tests. I change the property testing function to use the <code>genNumbers</code> generated values in the test:</p>
<pre class="brush: haskell">
prop_convertNumber :: Property
prop_convertNumber =
forAll genNumbers
(\x ->
(convertFromRoman . convertToRoman) x == x)
</pre>
<p>This works as expected when I test it in the REPL:</p>
<pre class="brush: shell">
λ> import Test.QuickCheck
λ> quickCheck prop_convertNumber
+++ OK, passed 100 tests.
</pre>
<p>I can now replace all my test with this function:</p>
<pre class="brush: haskell">
spec :: Spec
spec = do
describe "Converting to Roman Numerals" $ do
it "converts number to Roman and back" $ property $
prop_convertNumber
</pre>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/9c3a49bc1216d87cf833945a164883ab1c636d40">Commit Point</a></p>
<p>Hspec's QuickCheck wrapper provides a convenience function for scenarios like this: I can replace the <code>it</code> and <code>property</code> functions with <code>prop</code> like this (I need to add an import to make this work):</p>
<pre class="brush: haskell">
import Test.Hspec.QuickCheck (prop)
...
spec :: Spec
spec = do
prop "converts number to Roman and back" $
prop_convertNumber
</pre>
<p>When I run the tests, they should all pass, and QuickCheck reports back that it generated 100 tests for me.</p>
<pre class="brush: shell">
RomanNumerals
converts number to Roman and back
+++ OK, passed 100 tests.
Finished in 0.0017 seconds
1 example, 0 failures
</pre>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/1cb16e3bfe61eee4ab7f8c03f682dec338bbd848">Commit Point</a></p>
<p>It's great that QuickCheck generates the numbers and runs these tests, however, I would like to look under the hood and see how these conversions are working. I'd like to eyeball what the converted Roman numbers are looking like.</p>
<p>This is all pure code, I can't just print numbers in the terminal. I also don't want to make pure code dirty with IO. <code>Debug.Trace</code> is the solution.</p>
<p>Check this out:</p>
<pre class="brush: haskell">
import Debug.Trace
...
prop_convertNumber :: Property
prop_convertNumber =
forAll genNumbers
(\x ->
traceShow("number: ", (x, convertToRoman x)) $
(convertFromRoman . convertToRoman) x == x)
</pre>
<p><code>traceShow</code> will print out both the Arabic and the converted Roman number.</p>
<p>Here is a sample of what I received:</p>
<pre class="brush: shell">
...
("number: ",(825,"XCXCXCXCXCXCXCXCXCXV"))
("number: ",(662,"XCXCXCXCXCXCXCXXXII"))
("number: ",(246,"XCXCLXVI"))
("number: ",(921,"XCXCXCXCXCXCXCXCXCXCXXI"))
converts number to Roman and back
+++ OK, passed 100 tests.
Finished in 0.0030 seconds
1 example, 0 failures
</pre>
<p>Oh, that 825 does not look like a valid Roman number. Of course: my conversion table only has values from 1 through 90. It can only convert numbers up to 98. I need to make sure QuickCheck will generate numbers within this range.</p>
<pre class="brush: haskell">
numbers :: [Int]
numbers = [1..98]
</pre>
<p>When I run the tests, I got lucky by QuickCheck, it used 98 for one of the conversions:</p>
<pre class="brush: shell">
...
("number: ",(98,"XCVIII"))
("number: ",(62,"LXII"))
("number: ",(54,"LIV"))
converts number to Roman and back
+++ OK, passed 100 tests.
Finished in 0.0034 seconds
1 example, 0 failure
</pre>
<p>All good, now!</p>
<p><a href="https://github.com/adomokos/roman-numerals-hs/commit/ef4a61866d8f9c75b0236aafd344c7618dddfb9c">Commit Point</a></p>
<p>QuickCheck has a little brother, called <a href="https://github.com/feuerbach/smallcheck">SmallCheck</a>, which will generate numbers up-to a defined depth. It might be a better fit for the Roman Numeral Kata, but that would be a different blog post. I encourage you to explore that library and see how you could change the test code to work with it.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-45113296149780052562018-10-13T12:33:00.000-05:002018-10-15T15:24:50.278-05:00Haskell by Day<p>I had played around with Clojure for a couple of years by 2015. I wanted to use it for more than just a toy-project, and when we had to build a data pipeline for my employer, I was more than happy to give it a try.<br>
I hired a developer for our team who had more serious functional programming background than I had. He started doing interesting things with our code base like separating out functions based on purity and using currying like no one I had seen before.</p>
<p>I asked him, how he knows so much about FP. His answer was plain and simple: "I've been learning Haskell." Oh. I have always thought Haskell is this far-out, esoteric, academic language that no one uses. I realized he knew more Clojure just by learning Haskell. Then he showed me Elm and I was blown away. (Elm has its roots in Haskell, it's a similar ML language.)</p>
<p>A couple of months later I watched Chad Fowler's <a href="https://www.youtube.com/watch?v=sAsRtZEGMMQ&t=6s">talk</a> about his adventures at WunderList in which he talked about a routing application that they wrote in less than fifty (50) lines of Haskell. It never went done and was blazing fast. His message (the one that I picked up anyway) was "Haskell, Haskell, learn Haskell".</p>
<p>I started digging into the language by following the examples from the book <a href="http://learnyouahaskell.com/">Learn You a Haskell for Great Good</a>. It was an easy read, I was cruising through the examples in GHCi (REPL). Then I stopped looking at the language for about 2-3 months as something came up. I went back to it and was shocked to realize that I did not remember anything. Zero. What the heck is this weird syntax <code>(x:xs)</code>?</p>
<p>I realized reading the book and following the examples in a REPL was not enough, I had to keep coming back to them, and most importantly, practice them. The author of the <a href="http://haskellbook.com/">HaskellProgramming book</a>, Chris Allen had a good talk on <a href="https://www.youtube.com/watch?v=Bg9ccYzMbxc">how to learn Haskell</a> in less than five years. The message is simple: learn by doing. And keep doing it. This is how my little learning project <a href="https://www.github.com/adomokos/haskell-katas">haskell-katas</a> was born. I needed something I could come back to and practice with. It had mostly helped, but I was still confused by its error messages and type classes, which made more sense by more reading and practice.</p>
<p>I liked my <a href="https://www.kennasecurity.com/">employer</a>, we worked on some interesting problems, but I wanted to try myself on the job market, see how marketable my skills are. At one point I had to solve a Tetris game challenge for a company as a take-home exercise. I knew they were interested in folks with functional thinking. I had no intention working on that problem in Ruby/Python/JavaScript, but I thought it would be interesting to come up with a solution in Haskell. They said I could do it in Haskell and I jumped on the task. I began working on it on a Friday afternoon, played with the problem for an hour or two, I tried to see how I could represent the data with Algebraic Data Types and maybe map the different shapes on the board. When that worked, I knew I'll be able to submit a solution. I worked on it over the weekend and by Monday morning I had the solution working. I submitted my code 3 days early and got invited for an in-person interview. This was the moment I realized I might be able to find a job working with Haskell.</p>
<p>Finding a role with Haskell got more important to me, as the one or two hours here and there in the evening or weekends will never be enough to learn the language fully. I found a job posting on StackOverflow that listed Haskell as a requirement, here in Chicago. I applied, and after a few rounds of interviews I received an offer from <a href="https://www.futurefinance.com/">them</a> which I accepted.</p>
<p>We are working on a student loan solution for the UK market. Haskell is the brain of the application, all the different calculation models and rules are coded in it. I was intrigued to work for the only company in Chicago that uses Haskell. I started my job about a month ago and boy, it's been an adventure. I feel very slow sometimes, but more productive when I submit a change that makes the code a bit easier to test. For example, I had to dive into a couple of days of <a href="https://github.com/adomokos/aeson-learning">learning</a> just to interact with JSON. All in all, I am super happy that I can do something by day that I learned at night, and with this new role I can take my functional programming skills to a new level.</p>
<p>You might ask: How about Ruby? Am I still using? Oh yeah, we have an API app in Rails, plus I started mentoring our intern by pairing with him on different challenges in Ruby a couple of times a week. We also have a fairly significant code base in Python that our data scientists put together to test the Haskell component. I enjoy going into there and help the team make the code better.</p>
<p>The future is polyglot. Follow your heart and work with the language you want to learn. That's the only way to get real deep into it!</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-18660726517299137442018-06-18T09:16:00.000-05:002018-06-18T09:16:56.473-05:00Haskell Stack vs. Stackage - Explained<p>I am sitting in the Denver Airport terminal, waiting for my flight to take me home from <a href="https://lambdaconf2018.dryfta.com/en/">Lambda Conf 2018</a> to Chicago. This conference was the best I have attended in recent years: the quality of the talks, the friendliness of the presenters and the attendees is just something I have not seen in a long time.</p>
<p>I set down next to a guy (<a href="https://github.com/DanBurton">Dan Burton</a> to be exact) with my lunch a couple of days ago and I started a conversation with him. When we chatted about open source, he mentioned he works on <a href="https://www.stackage.org">Stackage</a>. I heard about it, but I asked him to tell me more. I learned quite a bit, I figured others would benefit from having this information, hence the idea of this blog post was born.</p>
<p><a href="https://docs.haskellstack.org/en/stable/README/">Stack</a> came along a couple of years ago to save us all from "Cabal Hell". I never had to experience it as I started learning Haskell only two years ago, long after Stack and Stackage was born. But understanding what they provide helps me appreciate it even more.</p>
<p>When you start a new project with Stack, there is one line in <code>stack.yaml</code> that has significant importance:</p>
<pre class="brush: shell">
resolver: lts-11.11
</pre>
<p>This value makes the connection between Stack and Stackage.</p>
<p>
"<em>What is <code>lts-11.11</code>?</em>" - It's the long term support version of Stackage.<br>
"<em>What is Stackage then?</em>" - It's a set of Haskell tools and libraries tested together in a snapshot making sure that the specified versions work together.<br>
"<em>A snapshot? What's that?</em>" - An LTS or Nightly release of packages.<br>
"<em>Isn't this a lot of work? Testing all these libraries together…</em>" - Oh yes it is, but the good thing is that it’s automated for the most part.<br>
"<em>How many people are working on this?</em>" - <a href="https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md">Eight</a>, but there are also some devops people at <a href="https://www.fpcomplete.com/">FP Complete</a> that occasionally help with the server.<br>
"<em>How often are the libraries tested?</em>" - There is a nightly snapshot released (almost) every night. There is an LTS snapshot minor bump (e.g. lts-11.11 -> lts-11.12) released (almost) every week. LTS major releases (e.g. lts-11 -> lts-12) are approximately every 3 to 5 months.<br>
"<em>Which one should I use?</em>" - the LTS snapshot of course. Unless you are curious and want to see how a library is changing daily.<br>
"<em>But I have GHC installed globally on my computer. Is that used?</em>" - It depends. If the LTS snapshot you specify in your project uses a different GHC version than what you have outside of Stack, that LTS specified GHC version will be installed.<br>
"<em>Give me an example!</em>" - Sure.</p>
<p>First, let's see what is installed globally. When I run <code>which ghc</code> this is what I get: <code>/usr/local/bin/ghc</code>. And when I peek into this file, I see it points to my homebrew installed ghc, with version 8.4.3:</p>
<pre class="brush: shell">
#!/bin/sh
exedir="/usr/local/Cellar/ghc/8.4.3/lib/ghc-8.4.3/bin"
exeprog="ghc-stage2"
executablename="$exedir/$exeprog"
datadir="/usr/local/Cellar/ghc/8.4.3/share"
bindir="/usr/local/Cellar/ghc/8.4.3/bin"
topdir="/usr/local/Cellar/ghc/8.4.3/lib/ghc-8.4.3"
executablename="$exedir/ghc"
exec "$executablename" -B"$topdir" ${1+"$@"}
</pre>
<p>Now when I run <code>ghc-pkg list</code>, I see 33 packages installed with this system-level GHC version:</p>
<pre class="brush: shell">
% ghc-pkg list
/usr/local/Cellar/ghc/8.4.3/lib/ghc-8.4.3/package.conf.d
Cabal-2.2.0.1
array-0.5.2.0
base-4.11.1.0
binary-0.8.5.1
...
</pre>
<p>I have not installed any packages myself into this GHC version, all those 33 packages come with GHC.</p>
<p>I have a project where the resolver is <code>lts-11.11</code>. When I run <code>stack exec -- ghc-pkg list</code> in this project (after it was successfully built of course), the following libraries are listed. I left out the bulk of the libraries, as the key point here is the different layers and not what is in those:</p>
<pre class="brush: shell">
% stack exec -- ghc-pkg list
/Users/adomokos/.stack/programs/x86_64-osx/ghc-8.2.2/lib/ghc-8.2.2/package.conf.d
Cabal-2.0.1.0
array-0.5.2.0
base-4.10.1.0
...
/Users/adomokos/.stack/snapshots/x86_64-osx/lts-11.11/8.2.2/pkgdb
Cabal-2.0.1.1
HUnit-1.6.0.0
StateVar-1.1.1.0
aeson-1.2.4.0
...
/Users/adomokos/Projects/persistent-test/.stack-work/install/x86_64-osx/lts-11.11/8.2.2/pkgdb
katip-0.5.5.1
persistent-test-0.1.0.0
</pre>
<p>The 3 paths listed above in this shell snippet is where Haskell packages are pulled from:</p>
<ol>
<li>Global - the system-level GHC packages list, Stack will never install anything into this</li>
<li>Snapshot - a database shared by all projects using the same snapshot</li>
<li>Local - Project specific database</li>
</ol>
<p>But wait! What is GHC 8.2.2 doing there? I have version 8.4.3 installed at the system level. As it turns out, Stack, based on the LTS information uses a different version of GHC. I have GHC version 8.4.3 at the system level, but LTS-11.11 uses GHC version 8.2.2.</p>
<p>Let’s prove that out further:</p>
<pre class="brush: shell">
% stack exec -- which ghc
/Users/adomokos/.stack/programs/x86_64-osx/ghc-8.2.2/bin/ghc
% stack exec -- ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.2.2
</pre>
<p>Ha, Stack rolls its own GHC version and ignores the system-level version if it's different than what it needs. How cool is that!</p>
<p>When I went to Stackage's website, I noticed that a newer version of LTS was released recently. I had <a href="https://www.stackage.org/lts-11.11">LTS-11.11</a> (released on 05/28/2018), but the latest version is (as of this writing) <a href="https://www.stackage.org/lts-11.13">LTS-11.13</a> (released on 06/09/2018). I updated <code>stack.yaml</code> to use the newer version and rebuilt the project. Ran the app and everything worked properly.</p>
<p>What changed between the two LTS snapshots? <a href="http://Stackage.org">Stackage.org</a> has a very good comparison page, <a href="https://www.stackage.org/diff/lts-11.11/lts-11.13">this is</a> where you can follow the diffs. It seems not many of the packages changed that I used, however, <code>postgresql-simple</code> went from 0.5.3.0 to 0.5.4.0. Since LTS-11.13 is specified in <code>stack.yaml</code> and that LTS needs <code>postgresql-simple</code> version 0.5.4.0, what happens when I specify version 0.5.3.0 in package.yaml?</p>
<p>I changed <code>package.yaml</code> this way:</p>
<pre class="brush: shell">
dependencies:
- base >= 4.7 && < 5
- postgresql-simple == 0.5.3.0
...
</pre>
<p>When I ran stack build, this friendly error message let me know that I'd like to use a version of a package that is not in the provided LTS snapshot:</p>
<pre class="brush: shell">
Error: While constructing the build plan, the following exceptions were encountered:
In the dependencies for persistent-test-0.1.0.0:
postgresql-simple-0.5.4.0 from stack configuration
does not match ==0.5.3.0 (latest matching version
is 0.5.3.0)
needed since persistent-test is a build target.
Some different approaches to resolving this:
* Set 'allow-newer: true' to ignore all version
constraints and build anyway.
* Consider trying 'stack solver', which uses the cabal-install
solver to attempt to find some working build
configuration. This can be convenient when dealing with many
complicated constraint errors, but results
may be unpredictable.
* Recommended action: try adding the following to your extra-deps
in /Users/adomokos/Projects/persistent-test/stack.yaml:
- postgresql-simple-0.5.3.0
Plan construction failed.
</pre>
<p>Once I removed the version specification for the <code>postgresql-simple</code> package, it built successfully. But did it pick the correct version since I did not specify it?</p>
<pre class="brush: shell">
% stack exec -- ghc-pkg list | grep postgresql-simple
postgresql-simple-0.5.4.0
</pre>
<p>Yep, the correct, Stackage LTS-11.13 version was in fact installed.</p>
<p>I grabbed all the package names from LTS-11.13, I counted 2474 packages that got tested against each other for this particular LTS release. Kudos to the Stackage Curator <a href="https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md">Team</a> for making sure we will only use packages that are playing nice with each other!</p>
<p>(Thanks to Dan for proofreading my post, this writing is more accurate with his feedback.)</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-33042128462383210252018-04-19T08:57:00.000-05:002018-04-19T08:57:38.407-05:00Path Count<p>I've bumped into this brain teaser recently:</p>
<p>"Given two integer numbers describing an n by m graph, where n represents the height and m represents the width, calculate the number of ways you can get from the top left to the bottom right if you can only go right and down"</p>
<p>That's a fine challenge, and prior to knowing recursive types in Haskell, I don’t know how I would have approached the problem.</p>
<p>Let's draw what the paths would look like first.</p>
<p>Given 1x1, the case is pretty simple:</p>
<pre class="brush: haskell">
{-
The is what the matrix would look like:
(0,1) - (1,1)
| |
(0,0) - (1,0)
-}
</pre>
<p>The only way to get from the top left to the bottom right is to "walk" the perimeters:</p>
<pre class="brush: haskell">
{-
(0,1) - (1,1) - (1,0)
(0,1) - (0,0) - (1,0)
-}
</pre>
<p>This case is so easy, I don’t even bother with devising a solution. Let's look at a 1 by 2 graph:</p>
<pre class="brush: haskell">
{-
(0-1) - (1,1) - (2,1)
| | |
(0-0) - (1,0) - (2,0)
-}
</pre>
<p>Following the rules laid out, there are 3 ways to get from the top left to the bottom right point:</p>
<pre class="brush: haskell">
{-
(0-1) - (1,1) - (2,1) - (2,0)
(0-1) - (1,1) - (1,0) - (2,0)
(0-1) - (0,0) - (1,0) - (2,0)
-}
</pre>
<p>The rule "you can only go right and down" tells us something: it's a binary tree. How could I draw up a recursive tree structure for this?</p>
<p>I'd like to make sure the logic is correct, I put all this logic into an HSpec file. How 'bout this?</p>
<pre class="brush: haskell">
-- test/PathCountSpec.hs
import Test.Hspec
main :: IO ()
main = hspec spec
type Point = (Int, Int)
data Tree a = Leaf
| Node a (Tree a) (Tree a)
deriving (Show, Eq)
spec :: Spec
spec =
describe "Path Count" $ do
it "can calculate tree branches for 1x2 matrix" $ do
let tree =
Node (0,1)
(Node (1,1)
(Node (2,1) Leaf
(Node (2,0) Leaf Leaf))
(Node (1,0) (Node (2,0) Leaf Leaf)
Leaf))
(Node (0,0)
(Node (1,0)
(Node (2,0) Leaf Leaf)
Leaf)
Leaf)
{-
Possible paths:
(0,1) - (1,1) - (2,1) - (2,0)
(0,1) - (1,1) - (1,0) - (2,0)
(0,1) - (0,0) - (1,0) - (2,0)
-}
pending
</pre>
<p>The key here, that the number of times <code>Node (2,0) Leaf Leaf</code> appears is the number of different ways I can get from the top left to the bottom right. All I have to do is counting the number of times this sub-tree appears in the tree itself.</p>
<p>I wrote a function (that I put into the HSpec file itself) to do just that:</p>
<pre class="brush: haskell">
leafCount :: Tree Point -> Int
leafCount Leaf = 0
leafCount (Node _ Leaf Leaf) = 1
leafCount (Node _ left right) = leafCount left + leafCount right
</pre>
<p>When I call this function with the provided tree I have in the spec, I should receive 3. This assertion passes:</p>
<pre class="brush: haskell">
leafCount tree `shouldBe` 3
</pre>
<p>I manually had to build up this tree, next I looked at how I could generate it myself based on the top left and bottom right points. I had to make sure I won’t add branches outside of the matrix, what I accomplished with two guards.</p>
<pre class="brush: haskell">
buildTree :: Point -> Point -> Tree Point
buildTree (a,b) end@(c,d)
| a > c = Leaf
| b < 0 = Leaf
| otherwise = Node (a, b) (buildTree (a+1,b) end) (buildTree (a,b-1) end)
</pre>
<p>This logic will keep "walking" right and down until the guards will stop it.</p>
<p>I added another assertion to make sure this still passes:</p>
<pre class="brush: haskell">
buildTree (0,1) (2,0) `shouldBe` tree
</pre>
<p>I wanted to make sure this logic still holds when I pivot the two numbers. This is the next assertion I added:</p>
<pre class="brush: haskell">
buildTree (0,2) (1,0) `shouldBe` tree
</pre>
<p>Tests are passing. All right, I need to set up an outer function that takes two numbers and returns the number of paths.</p>
<p>Here it is:</p>
<pre class="brush: haskell">
pathCount :: Int -> Int -> Int
pathCount m n = leafCount $ fromTree m n
where fromTree m n = buildTree (0,m) (n,0)
</pre>
<p>This assertion will exercise this function:</p>
<pre class="brush: haskell">
pathCount 1 2 `shouldBe` 3
</pre>
<p>Everything is green!</p>
<p>I added one more test to make sure the 1x1 graph works:</p>
<pre class="brush: haskell">
pathCount 1 1 `shouldBe` 2
</pre>
<p>And finally I added a 2x2 test as well. This one has 6 different paths to get from the top left to the bottom right:</p>
<pre class="brush: haskell">
{-
Possible paths:
(0,2) - (1,2) - (2,2) - (2,1) - (2,0)
(0,2) - (1,2) - (1,1) - (2,1) - (2,0)
(0,2) - (1,2) - (1,1) - (1,0) - (2,0)
(0,2) - (0,1) - (1,1) - (2,1) - (2,0)
(0,2) - (0,1) - (1,1) - (1,0) - (2,0)
(0,2) - (0,1) - (0,0) - (1,0) - (2,0)
-}
pathCount 2 2 `shouldBe` 6
</pre>
<p>And when I run the spec again, it all works! You can find the solution in <a href="https://gist.github.com/adomokos/fd26a07d8c19f96905828a0670029f5b">this gist</a>.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-74952609398059400422018-01-28T14:35:00.000-06:002018-01-28T14:35:31.148-06:00Haskell to MySQL via YeshQL (Part 3.)<p>In the <a href="http://www.adomokos.com/2017/12/haskell-to-mysql-via-yeshql-part-2.html">previous</a> blog post we built a console app in Haskell that talks to MySQL via <a href="https://github.com/tdammers/yeshql">YeshQL</a>. It creates a client record and counts them with a SQL query.<br>
In the final part in this series, we will add automated tests to our application, we will save a user record along with its parent client and make sure all the saving happens in one unit of work.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/275d45ad1f6abe1f6f5eccb0e67c552543c96c90">This is</a> the commit point where we left it at the end of Part 2.</p>
<h4>Add Tests</h4>
<p>Let’s set up the great testing tool <a href="https://hspec.github.io/">HSpec</a> in our project.</p>
<p>First, replace the content of <code>test/Spec.hs</code> file with this:</p>
<pre class="brush: haskell">
{-# OPTIONS_GHC -F -pgmF hspec-discover #-}
</pre>
<p>This will let auto-discover any spec files we have in the <code>test</code> directory.</p>
<p>Let's add the first test to the project in the <code>test/Hashmir/DataSpec.hs</code> file:</p>
<pre class="brush: haskell">
module Hashmir.DataSpec where
import Test.Hspec
main :: IO ()
main = hspec spec
spec :: Spec
spec = do
describe "Hashmir Data" $ do
it "runs a test" $ do
True `shouldBe` True
</pre>
<p>We are not testing anything real here, we just want to make sure all the building blocks are in place.</p>
<p>Add the <code>test-suite</code> directive to <code>package.yml</code> file:</p>
<pre class="brush: javascript">
...
tests:
hashmir-test:
source-dirs: test/
main: Spec.hs
dependencies:
- hashmir
- hspec == 2.*
other-modules:
Hashmir.DataSpec
</pre>
<p><code>make build</code> should recompile the app, and <code>stack test</code> will run the entire test suite.</p>
<p>When all is good, you should see this:</p>
<pre class="brush: shell">
Hashmir.Data
Hashmir Data
runs a test
Finished in 0.0010 seconds
1 example, 0 failures
</pre>
<p>It wouldn't save much typing, but I like navigating the projects I am working on from a Makefile, I added these changes to run the tests with <code>make test</code>:</p>
<pre class="brush: shell">
...
test: ## Run the specs
@stack test
.PHONY: help test
</pre>
<p><a href="https://github.com/adomokos/hashmir/commit/9f7c14e6aa518da44338b7426822173053ecf6c0">Commit point</a></p>
<h4>Verify Client Create Logic</h4>
<p>Creating a record in the database is easy, we already verified it when we ran the app. However, making this automated and repeatable shows some challenges. We need to make sure that every test cleans after itself in the DB. We could wrap each and every spec in a transaction and just roll it back, but that would be quite complex. Dropping and rebuilding the database is fast as it is. Sure, it's a couple of hundred milliseconds, but that is negligible for now.</p>
<p>HSpec provides before hooks, we will hook into that.</p>
<p>Let's change the <code>test/Hashmir/DataSpec.hs</code> like this:</p>
<pre class="brush: haskell; highlight: [4,10,11,14,16,17,18];">
module Hashmir.DataSpec where
import Test.Hspec
import System.Process
import qualified Hashmir.Data as D
main :: IO ()
main = hspec spec
resetDB :: IO ()
resetDB = callCommand "make build-db"
spec :: Spec
spec = before resetDB $ do
describe "Hashmir Data" $ do
it "creates a Client record" $ do
clientId <- D.insertClient "TestClient" "testclient"
clientId `shouldBe` 1
</pre>
<p>We call <code>resetDB</code> with every single spec, that function makes a system call to rebuild the DB.</p>
<p>When you try executing the test, stack tries to recompile the app, but it presents an error:</p>
<pre class="brush: haskell">
test/Hashmir/DataSpec.hs:4:1: error:
Failed to load interface for ‘System.Process’
It is a member of the hidden package ‘process-1.4.3.0’.
Perhaps you need to add ‘process’ to the build-depends in your .cabal file.
</pre>
<p>Oh-oh. We need to add the <code>process</code> package to our test-suite, let's modify the <code>package.yml</code> like this:</p>
<pre class="brush: javascript; highlight: [6]">
tests:
hashmir-test:
source-dirs: test/
main: Spec.hs
dependencies:
- process
- hashmir
- hspec == 2.*
other-modules:
Hashmir.DataSpec
</pre>
<p>After adding the <code>process</code> package, regenerating the cabal file, we can now run our first test successfully:</p>
<pre class="brush: shell">
Hashmir.Data
Hashmir Data
Dropping and rebuilding database hashmir_test
runs a test
Finished in 0.1378 seconds
1 example, 0 failures
</pre>
<p>The beauty of this solution is that we can run it over and over again, the test will pass as the checked <code>clientId</code> will always be 1, since the database is recreated every time.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/be20905a5ad5199c7b7b5e1eb4de8705a3656770">Commit point</a></p>
<h4>Add a User Record Along With Client</h4>
<p>Let's add a failing spec for this first. Add the following content to the <code>test/Hashmir/DataSpec.hs</code> file:</p>
<pre class="brush: haskell">
it "creates a Client and a User record" $ do
clientId <- D.insertClient "TestClient" "testclient"
userId <- D.insertUser clientId "joe" "joe@example.com" "password1"
userId `shouldBe` 1
</pre>
<p>There is no <code>insertUser</code> function, let's add it. We also need to add the SQL template to the YeshQL code. It's very similar to the Client insert script, here are all the changes for that:</p>
<pre class="brush: haskell; highlight: [9,10,11,12,13,14,15,16]">
[yesh|
-- name:countClientSQL :: (Int)
SELECT count(id) FROM clients;
;;;
-- name:insertClientSQL
-- :client_name :: String
-- :subdomain :: String
INSERT INTO clients (name, subdomain) VALUES (:client_name, :subdomain);
;;;
-- name:insertUserSQL
-- :client_id :: Integer
-- :login :: String
-- :email :: String
-- :password :: String
INSERT INTO users (client_id, login, email, password)
VALUES (:client_id, :login, :email, :password);
|]
</pre>
<p>And the <code>insertUser</code> function like this:</p>
<pre class="brush: haskell">
insertUser :: Integer -> String -> String -> String -> IO Integer
insertUser clientId login email password =
withConn $ insertUserSQL clientId login email password
</pre>
<p>When I run <code>make test</code>, this is the output printed on the screen:</p>
<pre class="brush: shell">
Hashmir.Data
Hashmir Data
Dropping and rebuilding database hashmir_test
creates a Client record
Dropping and rebuilding database hashmir_test
creates a Client and a User record
Finished in 0.2642 seconds
2 examples, 0 failures
</pre>
<p>The lines <code>Dropping and rebuilding database hashmir_test</code> is too much noise, let's remove it from the Makefile.</p>
<pre class="brush: shell">
Hashmir.Data
Hashmir Data
creates a Client record
creates a Client and a User record
Finished in 0.2354 seconds
2 examples, 0 failures
</pre>
<p>This looks much cleaner.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/63c976e618fe9e9b9ca1c833ad052f62a7d3486b">Commit point</a></p>
<h4>Roll Back Transactions When Error Occurs</h4>
<p>The happy path of our application is working well: the User and Client records are inserted properly. First, the Client is saved, its <code>id</code> is used for the User record to establish the proper references. But we should treat these two inserts as one unit of work: if the second fails, it should roll back the first insert.</p>
<p>Let's write a test for it. I'll make the created Client's <code>id</code> intentionally wrong by incrementing it by one.</p>
<pre class="brush: haskell">
it "rolls back the transaction when failure occurs" $ do
clientId <- D.insertClient "TestClient" "testclient"
_ <- D.insertUser (clientId + 1) "joe" "joe@example.com" "password1"
clientCount <- D.withConn $ D.countClientSQL
clientCount `shouldBe` Just 0
</pre>
<p>When I run the tests, this is the error I am getting:</p>
<pre class="brush: shell">
Hashmir.Data
Hashmir Data
creates a Client record
creates a Client and a User record
rolls back the transaction when failure occurs FAILED [1]
Failures:
test/Hashmir/DataSpec.hs:23:
1) Hashmir.Data, Hashmir Data, rolls back the transaction when failure occurs
uncaught exception:
SqlError (SqlError {seState = "",
seNativeError = 1452,
seErrorMsg = "Cannot add or update a child row: a foreign key constraint
fails (`hashmir_test`.`users`, CONSTRAINT `client_id`
FOREIGN KEY (`client_id`) REFERENCES `clients` (`id`))"})
Randomized with seed 668337839
Finished in 0.3924 seconds
3 examples, 1 failure
</pre>
<p>The database is protecting itself from an incorrect state, a User record won't be saved with an <code>id</code> that does not match a record in the <code>clients</code> table. This exception is justified, although, it could be handled better with a Maybe type, that's not the point right now. Let's just expect this exception for now to see a proper test failure.</p>
<p>Change the test like this:</p>
<pre class="brush: haskell">
it "rolls back the transaction when failure occurs" $ do
clientId <- D.insertClient "TestClient" "testclient"
(D.insertUser (clientId + 1)
"joe"
"joe@example.com"
"password1")
`shouldThrow` anyException
clientCount <- D.withConn $ D.countClientSQL
clientCount `shouldBe` Just 0
</pre>
<p>The spec now produces the error I would expect:</p>
<pre class="brush: shell">
Failures:
test/Hashmir/DataSpec.hs:31:
1) Hashmir.Data, Hashmir Data, rolls back the transaction when failure occurs
expected: Just 0
but got: Just 1
Randomized with seed 1723584293
Finished in 0.3728 seconds
3 examples, 1 failure
</pre>
<p>Finally, we have a spec that fails correctly, as we are not rolling back the created Client record.</p>
<p>The reason the Client record is not rolled back is that we use two different transactions to persist the records: first, the Client record is saved and the connection is committed, and then the User record is attempted to be saved. It fails, the record is not created, but the Client record has already been committed to the database. This is our problem, we should reuse the same connection for both save operations, and only commit it after the second one.</p>
<p>Let's refactor the code to do that. Both the <code>insertClient</code> and <code>insertUser</code> now accept a connection:</p>
<pre class="brush: haskell">
insertClient :: H.IConnection conn =>
String -> String -> conn -> IO Integer
insertClient name subdomain =
insertClientSQL name subdomain
insertUser :: H.IConnection conn =>
Integer -> String -> String -> String -> conn -> IO Integer
insertUser clientId login email password =
insertUserSQL clientId login email password
</pre>
<p>The specs now has to be modified to pass in the connection:</p>
<pre class="brush: haskell">
spec :: Spec
spec = before resetDB $ do
describe "Hashmir Data" $ do
it "creates a Client record" $ do
clientId <- D.withConn $ D.insertClient "TestClient" "testclient"
clientId `shouldBe` 1
it "creates a Client and a User record" $ do
userId <- D.withConn (\conn -> do
clientId <- D.insertClient "TestClient" "testclient" conn
D.insertUser clientId "joe" "joe@example.com" "password1" conn)
userId `shouldBe` 1
it "rolls back the transaction when failure occurs" $ do
(D.withConn (\conn -> do
clientId <- D.insertClient "TestClient" "testclient" conn
D.insertUser (clientId+1) "joe" "joe@example.com" "password1" conn))
`shouldThrow` anyException
clientCount <- D.withConn $ D.countClientSQL
clientCount `shouldBe` Just 0
</pre>
<p>And finally, the Main function has to be updated as well:</p>
<pre class="brush: haskell">
main :: IO ()
main = do
clientId <- D.withConn $ D.insertClient "TestClient" "testclient"
putStrLn $ "New client's id is " ++ show clientId
Just clientCount <- D.withConn D.countClientSQL
putStrLn $ "There are " ++ show clientCount ++ " records."
</pre>
<p>When you run the tests, they should all pass now.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/b488dad7c610a84ba84e90e0340d0449ac094d3a">Commit point</a></p>
<h4>Summary</h4>
<p>In this blog series we set up YeshQL, added logic to insert Client and its dependent User records, we added tests and made sure all the writes are in one transaction.</p>
<p>Our final solution works, but it requires the connection to be passed in. Using a <a href="https://hackage.haskell.org/package/mtl-2.2.1/docs/Control-Monad-Reader.html">Reader Monad</a> would be a more elegant solution, but that should be a different blog post.</p>
Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-54128072964978178982017-12-12T13:02:00.000-06:002018-01-30T20:57:44.890-06:00Haskell to MySQL via YeshQL (Part 2.)<p>In the <a href="http://www.adomokos.com/2017/11/haskell-to-mysql-via-yeshql-part-1.html">previous</a> blog post, we built the skeleton of a console app in Haskell that talks to MySQL via <a href="https://github.com/tdammers/yeshql">YeshQL</a>. We used a Makefile to rebuild the database, compile the code and run the app. <a href="https://github.com/adomokos/hashmir/commit/78a597e2c348abe751178812367f260fde69edb6">This is</a> the commit point where we left it at the end of Part 1.</p>
<p>In this article we will add functionality to create <code>clients</code> records, we will extract the logic that acquires, commits and closes the connection, and finally, we will move the database related code into its own module.</p>
<h4>Insert a Client record</h4>
<p>We can query the <code>clients</code> table, however, there are no records in that table. Let's add one script to the SQL template.</p>
<p>Modify the YeshQL templates by adding the following code:</p>
<pre class="brush: haskell;highlight: [4,5,6,7,8]">
[yesh|
-- name:countClientSQL :: (Int)
SELECT count(id) FROM clients;
;;;
-- name:insertClientSQL
-- :client_name :: String
-- :subdomain :: String
INSERT INTO clients (name, subdomain) VALUES (:client_name, :subdomain);
|]
</pre>
<p>Three semicolons are used to separate SQL statements. The <code>-- name</code> is used to provide a function name that we can refer to in our code. The two lines are the input arguments to the generated function. Those arguments are the values in the INSERT statement on the last line.</p>
<p>Let's add the following Haskell function to insert a new Client record:</p>
<pre class="brush: haskell">
insertClient :: String -> String -> IO ()
insertClient name subdomain = do
conn <- getConn
clientId <- insertClientSQL name subdomain conn
commit conn
disconnect conn
putStrLn $ "New client's id is " ++ show clientId
</pre>
<p>The generated <code>insertClientSQL</code> (note the name in the SQL template) is invoked with the two specified arguments plus the connection. The uncommitted connection will not write the record to the table, without that statement the <code>countClientSQL</code> function would return 0 records.<br>
Disconnecting a connection is a good practice (if you can't pool those connections), free up resources when you don't need them.</p>
<p>Invoke the created <code>insertClient</code> function from <code>main</code> function like this:</p>
<pre class="brush: haskell">
main = do
insertClient "TestClient" "testclient"
countClient
</pre>
<p>When I try to build the app (use <code>make build</code> for it), I get the following errors:</p>
<pre class="brush: shell">
/Haskell/hashmir/app/Main.hs:34:5: error:
Variable not in scope: commit :: Connection -> IO a0
</pre>
<p>It turns out that <code>Database.HDBC</code> library is needed to invoke <code>commit</code> and <code>disconnect</code>.<br>
Add the following import statement to the top of the file:</p>
<pre class="brush: haskell">
import qualified Database.HDBC as H
</pre>
<p>Update the function to use the qualified names for the <code>commit</code> and <code>disconnect</code> calls.</p>
<pre class="brush: haskell; highlight: [5,6];">
insertClient :: String -> String -> IO ()
insertClient name subdomain = do
conn <- getConn
clientId <- insertClientSQL name subdomain conn
H.commit conn
H.disconnect conn
putStrLn $ "New client's id is " ++ show clientId
</pre>
<p>The project should successfully build, when you run the app (use <code>make run</code> to do it), you should see this output:</p>
<pre class="brush: shell">
% make run
Dropping and rebuilding database hashmir_test
time ~/.local/bin/hashmir-exe
New client's id is 1
There are 1 records.
real 0m0.020s
user 0m0.009s
sys 0m0.006s
</pre>
<p>The database got rebuilt, one client record was inserted and the count function counted that record.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/dbdbcc3d41bf84f7230ef0ff98f78f6752c115ab">Commit point</a></p>
<h4>Introducing: withConn</h4>
<p>Committing and disconnecting a connection is generally a good practice. Let's match the <code>countClient</code> function with the <code>insertClient</code> function and call <code>commit</code> and <code>disconnect</code> the connection there as well.</p>
<pre class="brush: haskell; highlight: [5,6];">
countClient :: IO ()
countClient = do
conn <- getConn
Just (clientCount) <- countClientSQL conn
H.commit conn -- added line
H.disconnect conn -- added line
putStrLn $ "There are " ++ show clientCount ++ " records."
</pre>
<p>Now both the <code>countClient</code> and the <code>insertClient</code> has the same duplicated logic:</p>
<pre class="brush: haskell">
conn <- getConn
...
H.commit conn -- added line
H.disconnect conn -- added line
</pre>
<p>This reminds me of the use of <a href="https://hackage.haskell.org/package/base-4.10.1.0/docs/System-IO.html#v:withFile">withFile</a> from the IO module. <code>withFile</code> accepts a lambda where the <code>handle</code> is passed to it and the code in the lambda can use the provided <code>handle</code>. We need the same thing here, <code>withConn</code> would accept an active connection. Consider this function:</p>
<pre class="brush: haskell">
withConn :: (Connection -> IO b) -> IO b
withConn f = do
conn <- getConn
result <- f conn
H.commit conn
H.disconnect conn
return result
</pre>
<p>Our refactored <code>insertClient</code> function would look like this:</p>
<pre class="brush: haskell">
insertClient :: String -> String -> IO ()
insertClient name subdomain = do
clientId <-
withConn (\conn -> do
insertClientSQL name subdomain conn
)
putStrLn $ "New client's id is " ++ show clientId
</pre>
<p>When you build the project and run it, it should work without errors.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/841959b7da65baf8b5a351d2e06d5ae0525b511d">Commit point</a></p>
<p>Thanks to Haskell's currying, this function can be further simplified. No need to provide the input argument in the lambda:</p>
<pre class="brush: haskell; highlight: [4];">
insertClient :: String -> String -> IO ()
insertClient name subdomain = do
clientId <-
withConn (do insertClientSQL name subdomain)
putStrLn $ "New client's id is " ++ show clientId
</pre>
<p>This looks much better, but we can further simplify this code:</p>
<pre class="brush: haskell; highlight: [3];">
insertClient :: String -> String -> IO ()
insertClient name subdomain = do
clientId <- withConn $ do insertClientSQL name subdomain
putStrLn $ "New client's id is " ++ show clientId
</pre>
<p>Now that function is easy to read!</p>
<p><a href="https://github.com/adomokos/hashmir/commit/2dac1bc5f175d6e14bd040ffe5a0858042a688ea">Commit point</a></p>
<p>Let's refactor the <code>countClient</code> function similarly.</p>
<pre class="brush: haskell">
countClient :: IO ()
countClient = do
Just (clientCount) <- withConn countClientSQL
putStrLn $ "There are " ++ show clientCount ++ " records."
</pre>
<p><a href="https://github.com/adomokos/hashmir/commit/3bb07613665dad7e07bd4bbb24b2c22fac981911">Commit point</a></p>
<h4>Extract Data Access Logic</h4>
<p>Having code that prints information on the screen from the functions that talks with the database makes them mixed with responsibilities. They should only return primitives, and the caller main function should print the reports. Let's make those functions a bit more clean:</p>
<pre class="brush: haskell">
insertClient :: String -> String -> IO Integer
insertClient name subdomain = do
withConn $ do insertClientSQL name subdomain
countClient :: IO (Maybe Int)
countClient = do withConn countClientSQL
</pre>
<p>I really like how Haskell makes the functions that uses IO impure or dirty: once they have been tainted, they are tainted. You should always try to isolate functions that are tainted from the pure ones as the pure ones are easier test - they have no side effects.</p>
<p>The <code>main</code> function is now responsible for reporting the result:</p>
<pre class="brush: haskell">
main :: IO ()
main = do
clientId <- insertClient "TestClient" "testclient"
putStrLn $ "New client's id is " ++ show clientId
Just clientCount <- countClient
putStrLn $ "There are " ++ show clientCount ++ " records."
</pre>
<p><a href="https://github.com/adomokos/hashmir/commit/f2b2393b437c32b669984d52f8195b1b9d643f95">Commit point</a></p>
<h4>Move Data Logic Into a Library</h4>
<p>Our <code>Main.hs</code> file has all the application logic. It looks pretty solid, let's move it into a library module. Copy all database access related code from <code>app/Main.hs</code> into <code>src/Hashmir/Data.hs</code> file like this:</p>
<pre class="brush: haskell">
{-#LANGUAGE TemplateHaskell #-}
{-#LANGUAGE QuasiQuotes #-}
module Hashmir.Data where
import Database.YeshQL
import qualified Database.HDBC as H
import Database.HDBC.MySQL
[yesh|
-- name:countClientSQL :: (Int)
SELECT count(id) FROM clients;
;;;
-- name:insertClientSQL
-- :client_name :: String
-- :subdomain :: String
INSERT INTO clients (name, subdomain) VALUES (:client_name, :subdomain);
|]
getConn :: IO Connection
getConn = do
connectMySQL defaultMySQLConnectInfo {
mysqlHost = "localhost",
mysqlDatabase = "hashmir_test",
mysqlUser = "hashmir_user",
mysqlPassword = "shei7AnganeihaeF",
mysqlUnixSocket = "/tmp/mysql.sock"
}
withConn :: (Connection -> IO b) -> IO b
withConn f = do
conn <- getConn
result <- f conn
H.commit conn
H.disconnect conn
return result
insertClient :: String -> String -> IO Integer
insertClient name subdomain =
withConn $ insertClientSQL name subdomain
countClient :: IO (Maybe Int)
countClient = withConn countClientSQL
</pre>
<p>This is the same logic we had in the <code>app/Main.hs</code> file, but now it is in the <code>Hashmir.Data</code> module.</p>
<p>The <code>Main</code> module becomes small once we remove all the code we just moved out of it:</p>
<pre class="brush: haskell">
module Main where
import qualified Hashmir.Data as D
main :: IO ()
main = do
clientId <- D.insertClient "TestClient" "testclient"
putStrLn $ "New client's id is " ++ show clientId
Just clientCount <- D.countClient
putStrLn $ "There are " ++ show clientCount ++ " records."
</pre>
<p>We also have to tell Cabal where it can find this code. Change the <code>Lib</code> directive to <code>Hashmir.Data</code> in the <code>package.yaml</code>:</p>
<pre class="brush: shell; highlight: [4];">
library:
source-dirs: src/
exposed-modules:
- Hashmir.Data
</pre>
<p>The project should build and when you run the app it should insert and return the number of <code>clients</code> records:</p>
<pre class="brush: shell">
% make run
Dropping and rebuilding database hashmir_test
time ~/.local/bin/hashmir-exe
New client's id is 1
There are 1 records.
real 0m0.022s
user 0m0.010s
sys 0m0.007s
</pre>
<p><a href="https://github.com/adomokos/hashmir/commit/275d45ad1f6abe1f6f5eccb0e67c552543c96c90">Commit point</a></p>
<p>This last change wraps up our Part 2 in this series. We can now create <code>clients</code> records, count them with a simple <code>withConn</code> function that properly opens, commits and closes the connection.</p>
<p>In the <a href="http://www.adomokos.com/2018/01/haskell-to-mysql-via-yeshql-part-3.html">third post</a> in this series, I will show you how we can insert two records in one transaction, how we can deal with errors and how this logic can be tested.</p>
Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-67187295319098134952017-11-30T03:54:00.000-06:002018-01-31T09:34:11.773-06:00Haskell to MySQL via YeshQL (Part 1.)<p>As I was looking for an easy way to talk to Postgres from Clojure, I discovered <a href="https://github.com/krisajenkins/yesql">Yesql</a>. I wanted to find something similar in Haskell, and I found <a href="https://github.com/tdammers/yeshql">YeshQL</a>. It's a template parsing library on top of <a href="https://github.com/ryantm/hdbc-mysql">HDBC</a>, exactly what I needed to keep SQL and Haskell code separate.</p>
<p>This blog post will show you how you can easily get YeshQL up and running and run queries against MySQL. I'll build a simple CRUD console app that you can follow along, I'll list commit points at the end of each section.</p>
<p>The Clojure tutorial I created for my blog posts is named <a href="https://github.com/adomokos/kashmir">Kashmir</a>, I'll name this project <a href="https://github.com/adomokos/hashmir">Hashmir</a>.</p>
<p>You will need <a href="https://docs.haskellstack.org/en/stable/README/">stack</a> and ghc installed, I have stack <code>Version 1.5.1 x86_64 hpack-0.17.1</code> and ghc version 8.0.2. MySQL is also needed, I have version 5.7.20, but I won't use anything fancy as far as the database goes, if you have MySQL installed, I am sure that will do it.</p>
<h4><a id="Generate_the_project_with_stack_9"></a>Generate the project with stack</h4>
<p>Generate the project with this command: <code>stack new hashmir</code>. Go inside the directory, build and deploy the app with <code>stack build && stack install</code>. Run the skeleton app:</p>
<pre class="brush: shell">
% ~/.local/bin/hashmir-exe
someFunc
</pre>
<p><a href="https://github.com/adomokos/hashmir/commit/de07eebdf3b0f40b550279a58231603eef4f4809">Commit point</a></p>
<h4>Using hpack</h4>
<p>Since I <a href="https://academy.mondaymorninghaskell.com/p/your-first-haskell-project">learned</a> about <a href="https://github.com/sol/hpack">hpack</a>, I never touch a Haskell project's cabal file any more. This blog post assumes you are familiar with this tool, feel free to learn about hpack more before you proceed.</p>
<p>Add this <code>package.yaml</code> to the project's root directory:</p>
<pre class="brush: shell">
name: hashmir
version: 0.1.0.0
author: Attila Domokos <adomokos@gmail.com>
maintainer: adomokos@gmail.com
copyright: 2017 Attila Domokos
category: Console App
homepage: https://github.com/adomokos/hashmir#readme
ghc-options: -Wall
dependencies:
- base >= 4.7 && < 5
library:
source-dirs: src/
exposed-modules:
- Lib
executables:
hashmir-exe:
source-dirs: app/
main: Main.hs
dependencies:
hashmir
ghc-options: -threaded -rtsopts -with-rtsopts=-N
</pre>
<p>Feel free to use your name and Github repo in this file.<br>
Delete and regenerate the project's cabal file with this command: <code>rm -f hashmir.cabal && stack build</code>. <code>stack install</code> should produce the same executable file.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/873bacf0c76787e9e199f994ca43d6de2f67cf3a">Commit point</a></p>
<h4><a id="Setting_up_the_Database_59"></a>Setting up the Database</h4>
<p>We will need a MySQL user and a database we can use in this project. Let's add a Makefile to script that out for us. You might need the master or root user in MySQL to create a role and grant access. If the Makefile target does not work, just log on to the MySQL console with your root account and add the role and grant access with the scripts you see in the Makefile.</p>
<p><a href="https://github.com/adomokos/hashmir/blob/master/resources/schema.sql">This is</a> the schema file we will work with. I won't include it in this post, but this script should drop and recreate the tables. Put it into the <code>./resources</code> directory.</p>
<p>These targets in the <a href="https://github.com/adomokos/hashmir/blob/c8fb2b66f07b0cb6cb79e7a22d4f7715218dd960/Makefile">Makefile</a> will create the role and will rebuild the database:</p>
<pre class="brush: shell">
create-db-user: ## Creates a DB user with the root MySQL user
mysql -u root --host $(HOST) -e "CREATE USER '$(DBUSER)'@'$(HOST)' IDENTIFIED BY '$(DBPASSWD)';" > /dev/null 2>&1
mysql -u root --host $(HOST) -e "GRANT ALL PRIVILEGES ON `$(DBNAME)`.* TO '$(DBUSER)'@'$(HOST)';" > /dev/null 2>&1
build-db: ## Builds the DB
@echo "Dropping and rebuilding database $(DBNAME)"
@mysql -u $(DBUSER) --password='$(DBPASSWD)' --host $(HOST) -e "DROP DATABASE IF EXISTS $(DBNAME);" > /dev/null 2>&1
@mysql -u $(DBUSER) --password='$(DBPASSWD)' --host $(HOST) -e "CREATE DATABASE $(DBNAME);" > /dev/null 2>&1
@mysql -u $(DBUSER) --password='$(DBPASSWD)' --host $(HOST) $(DBNAME) < resources/schema.sql > /dev/null 2>&1
</pre>
<p>You should be able to execute <code>make build</code> to rebuild the app, and <code>make run</code> to rebuild the DB and run the app:</p>
<pre class="brush: shell">
% make run
Dropping and rebuilding database crud_yeshql_test
time ~/.local/bin/hashmir-exe
someFunc
real 0m0.011s
user 0m0.002s
sys 0m0.007s
</pre>
<p><a href="https://github.com/adomokos/hashmir/commit/ff40249ac7bc767e9f57b52daccf690cbf3bae1a">Commit point</a></p>
<h4><a id="Writing_the_First_Query_95"></a>Writing the First Query</h4>
<p>There are two parts of using YeshQL's code:</p>
<ol>
<li>The SQL templates</li>
<li>Code that uses the generated functions from the template</li>
</ol>
<p>Modify the <code>app/Main.hs</code> file like this:</p>
<pre class="brush: haskell">
{-#LANGUAGE TemplateHaskell #-}
{-#LANGUAGE QuasiQuotes #-}
module Main where
import Database.YeshQL
import Database.HDBC.MySQL
[yesh|
-- name:countClientSQL :: (Int)
SELECT count(id) FROM clients;
|]
getConn :: IO Connection
getConn = do
connectMySQL defaultMySQLConnectInfo {
mysqlHost = "localhost",
mysqlDatabase = "hashmir_test",
mysqlUser = "hashmir_user",
mysqlPassword = "shei7AnganeihaeF",
mysqlUnixSocket = "/tmp/mysql.sock"
}
countClient :: IO ()
countClient = do
conn <- getConn
Just (clientCount) <- countClientSQL conn
putStrLn $ "There are " ++ show clientCount ++ " records."
main :: IO ()
main = countClient
</pre>
<p>When you try to build the project (I conveniently use <code>make build</code>) one of these errors are displayed:</p>
<pre><code class="language-shell">Failed to load interface for 'Database.YeshQL'
</code></pre>
<p>I started referencing <code>Database.YeshQL</code>, however, I did not add that library to the project. This is where hpack is helpful, we only have to add it to the <code>package.yaml</code> file, that will generate a cabal file with the correct references.</p>
<p>Let's modify the <code>dependencies</code> section of package.yaml file like this:</p>
<pre class="brush: shell">
...
dependencies:
- base >= 4.7 && < 5
- yeshql
- HDBC
- HDBC-mysql
...
</pre>
<p>When I try to build the project, I get the following error:</p>
<pre class="brush: shell">
Error: While constructing the build plan, the following exceptions were encountered:
In the dependencies for hashmir-0.1.0.0:
yeshql must match -any, but the stack configuration has no specified version (latest applicable is 3.0.1.3)
needed since hashmir-0.1.0.0 is a build target.
Recommended action: try adding the following to your extra-deps in ~/hashmir/stack.yaml:
- yeshql-3.0.1.3
</pre>
<p>By modifying the <code>extra-deps: []</code> array like this in <code>stack.yaml</code></p>
<pre class="brush: shell">
extra-deps: [
yeshql-3.0.1.3
]
</pre>
<p>will download YeshQL for the project and build it successfully.</p>
<p>When I run the app (use <code>make run</code> to do it), this is what I see:</p>
<pre class="brush: shell">
% make run
Dropping and rebuilding database hashmir_test
time ~/.local/bin/hashmir-exe
There are 0 records.
real 0m0.019s
user 0m0.009s
sys 0m0.006s
</pre>
<p>YeshQL generated a function on-the-fly, named <code>countClientSQL</code>. That's the function I invoked in the <code>countClient</code> function. Since there are no records in the table, 0 was returned from the function.</p>
<p><a href="https://github.com/adomokos/hashmir/commit/78a597e2c348abe751178812367f260fde69edb6">Commit point</a></p>
<p>We set up the project, ran the first query against MySQL from Haskell via YeshQL templates. This brings us to the end of part 1 of this series. In the <a href="http://www.adomokos.com/2017/12/haskell-to-mysql-via-yeshql-part-2.html">next article</a>, we'll start adding more SQL queries to insert and query various records.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-62277138036346268922017-08-25T08:08:00.000-05:002017-08-25T12:10:48.126-05:00String Calculator with Applicative Functors in Haskell<p>I like the simplicity of the <a href="http://osherove.com/tdd-kata-1/">String Calculator kata</a>. It's a typical example of the map-reduce algorithm, where the string has to be split by a delimiter, mapped into a list of integers and then reduced to their sum. I have often <a href="https://github.com/adomokos/stringcalulator_js_starter_kit">used it</a> as an example to quickly evaluate engineering candidates, try new languages and tools. This was the first challenge I tried to solve in Haskell about a year ago. I found the code the other day, I wanted to see how I could improve upon it with everything I've learned so far.</p>
<p>This is what I found from a year before:</p>
<pre class="brush: haskell">
import Test.Hspec
import Data.List.Split (splitOn)
calculator :: String -> Integer
calculator x = sum $ coerceString x ","
coerceString :: String -> String -> [Integer]
coerceString x c = map (coerce) (splitOn c x)
where
coerce "" = 0
coerce a = read a :: Integer
main :: IO ()
main = hspec $ do
describe "String Calculator" $ do
it "returns 0 for empty string" $ do
calculator "" `shouldBe` 0
it "returns 1 for '1'" $ do
calculator "1" `shouldBe` 1
it "returns 3 for '1,2,3'" $ do
calculator "1,2,3" `shouldBe` 6
</pre>
<p>This works for simple cases, but what should I do when there is a non-numeric string in the input arguments?</p>
<pre class="brush: haskell">
it "returns 0 for '1,2,!'" $ do
calculator "1,2,!" `shouldBe` 0
</pre>
<p>The algorithm crashes as I suspected:</p>
<pre class="brush: shell">
String Calculator
returns 0 for empty string
returns 1 for '1'
returns 3 for '1,2,3'
returns 0 for '1,2,!' FAILED [1]
Failures:
string_calculator_01.hs:22:
1) String Calculator returns 0 for '1,2,!'
uncaught exception: ErrorCall (Prelude.read: no parse)
Randomized with seed 1943023196
Finished in 0.0023 seconds
4 examples, 1 failure
</pre>
<p>I could easily wrap the entire logic and return zero when an exception occurs, however, Haskell can do better. Much better.</p>
<p>I can return a <code>Maybe Int</code> from the operation that parses the string to an integer: if the result is <code>Nothing</code> here, the overall result will be <code>Nothing</code>.<br>
There is a string parser in the <code>Text.Read</code> module that does just that, it's called <code>readMaybe</code>.</p>
<p>Here is how it works:</p>
<pre class="brush: shell">
% ghci
GHCi, version 8.0.2: http://www.haskell.org/ghc/ :? for help
Loaded GHCi configuration from /Users/adomokos/.ghci
λ> import Text.Read
λ> (readMaybe "12") :: Maybe Int
Just 12
λ> (readMaybe "!") :: Maybe Int
Nothing
λ>
</pre>
<p>I need to parse the list of strings, which is achieved by mapping over the values with <code>readMaybe</code>:</p>
<pre class="brush: shell">
GHCi, version 8.0.2: http://www.haskell.org/ghc/ :? for help
Loaded GHCi configuration from /Users/adomokos/.ghci
λ> import Text.Read (readMaybe)
λ> import Data.List.Split (splitOn)
λ> let xs = "1,2,3"
λ> splitOn (",") xs
["1","2","3"]
λ> map (\x -> readMaybe x :: Maybe Int) $ splitOn (",") xs
[Just 1,Just 2,Just 3]
</pre>
<p>I now have a list of Maybe Int values that can be reduced into a single Maybe Int value. Reducing an array of numbers would be super easy (<code>foldl (+) 0 [1,2,3]</code> or <code>foldl1 (+) [1,2,3]</code> or just simply <code>sum [1,2,3]</code>). It's obvious that I will need something similar.</p>
<p>Adding a number to a <code>Maybe Int</code> can be achieved with a Functor:</p>
<pre class="brush: shell">
λ> fmap (+10) (Just 4)
Just 14
λ> (+10) `fmap` (Just 4)
Just 14
λ> (+10) <$> Just 4
Just 14
</pre>
<p>All three expressions mean the same thing. The first is using the fmap as a conventional function name and arguments style, the second one uses the infix version of <code>fmap</code> and the third one is using a symbol.</p>
<p>This works for adding a number to a Maybe Int, however, I need to use an Applicative Functor to calculate the sum of two Maybe Ints.</p>
<pre class="brush: shell">
λ> (+) <$> Just 10 <*> Just 4
Just 14
</pre>
<p>Using Applicative Functors, folding the list of Maybe Ints happens like this:</p>
<pre class="brush: shell">
λ> let xs = [Just 1, Just 2, Just 3]
λ> foldl (\acc x -> (+) <$> x <*> acc) (Just 0) xs
Just 6
λ> foldl1 (\acc x -> (+) <$> x <*> acc) xs
Just 6
</pre>
<p>The solution now works, although a bit hard to read:</p>
<pre class="brush: haskell">
import Test.Hspec
import Text.Read (readMaybe)
import Data.List.Split (splitOn)
calculator :: String -> Maybe Int
calculator input =
foldr (\x acc -> (+) <$> x <*> acc) (Just 0) $
map (\x -> readMaybe x) $ splitOn "," input
main :: IO ()
main = hspec $ do
describe "String Calculator" $ do
it "returns Nothing for empty string" $ do
calculator "" `shouldBe` Nothing
it "returns Just 1 for '1'" $ do
calculator "1" `shouldBe` Just 1
it "returns Just 3 for '1,2,3'" $ do
calculator "1,2,3" `shouldBe` Just 6
it "returns Nothing for '1,2,3,!'" $ do
calculator "1,2,3,!" `shouldBe` Nothing
</pre>
<p>It's more meaningful after I've refactored it into chunks:</p>
<pre class="brush: haskell">
calculator :: String -> Maybe Int
calculator input =
let foldingFn acc x = (+) <$> acc <*> x
parsedInts = map (\x -> readMaybe x) . splitOn (",")
in foldr1 (foldingFn) (parsedInts input)
</pre>
<p>The <code>parsedInts</code> function can be further simplified:</p>
<pre class="brush: haskell">
calculator :: String -> Maybe Int
calculator input =
let foldingFn acc x = (+) <$> acc <*> x
parsedInts = map (readMaybe) . splitOn (",")
in foldr1 (foldingFn) (parsedInts input)
</pre>
<p>And finally, the sum of list of Maybe values can be calculated by using the <code>sequence</code> function like this:</p>
<pre class="brush: haskell">
calculator :: String -> Maybe Int
calculator input =
let parsedInts = map (readMaybe) . splitOn (",")
in fmap sum . sequence $ parsedInts input
</pre>
<p>I find this form to be the most readable, but I liked the journey of getting there through the Applicative Functors.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-31672560627371414522017-06-28T16:55:00.002-05:002017-06-28T16:55:54.014-05:00Scenario Outline in RSpec<p>Some business logic is best described in a table format. The rows and columns can better detail all the different permutations, than words and sentences. Gherkin handles this well with its <a href="https://github.com/cucumber/cucumber/wiki/Scenario-Outlines">scenario outlines</a>, however, RSpec does not have such feature.</p>
<p>As I wanted to test all the different permutation of a business logic, I ended up with a ~200 lines spec file. It was hard to review what use cases were covered as I had to scroll up and down to review the file. Had RSpec have a table format, this logic could have been described in 40 lines, which can easily fit into a screen.</p>
<p>It was obvious a simple DSL was needed, but what would that look like? This is what I came up with first:</p>
<pre class="brush: ruby">
RSpec.describe 'eating apples' do
def method_under_test(start, eat, left)
start - eat == left
end
def example_runner(examples)
lines = examples.split(/\n/)
lines.delete_at(0) # Don't need the title
lines.each do |line|
example = line.split('|')
start, eat, left = example.reject(&:blank?).map(&:to_i)
expect(method_under_test(start, eat, left)).to eq(true)
end
end
it 'can tell how many are remaining' do
examples =
<<~TABLE
| start | eat | left |
| 12 | 5 | 7 |
| 20 | 5 | 15 |
TABLE
example_runner(examples)
end
end
</pre>
<p>I am using the <code>example_runner</code> method to parse the examples table and run the specs one by one. The spec itself is clear, the examples TABLE is easy to visualize.</p>
<p>The spec documentation, however, does not reveal the used table. This is what it looks like:</p>
<pre class="brush: shell">
eating apples
can tell how many are remaining
</pre>
<p>Let’s say, one of the examples will trigger an error. When I change the second example’s “left” column’s 15 value to 16, this is what I see when I run the spec.</p>
<pre class="brush: shell">
F
Failures:
1) eating apples can tell how many are remaining
Failure/Error: expect(method_under_test(start, eat, left)).to eq(true)
expected: true
got: false
(compared using ==)
</pre>
<p>This is not very useful. I don’t know which example failed. Fortunately RSpec lets us provide more context for an expectation. When I change the line of the assertion with an added context like this,</p>
<pre class="brush: ruby">
expect(method_under_test(start, eat, left)).to eq(true), line
</pre>
<p>the example that failed is properly reported by RSpec:</p>
<pre class="brush: shell">
F
Failures:
1) eating apples can tell how many are remaining
Failure/Error: expect(method_under_test(start, eat, left)).to eq(true), line
| 20 | 5 | 16 |
</pre>
<p>This could be a decent solution, however, not printing the examples for the documentation output has major drawbacks. Every row in the table should be a separate example, it should be expressed as such. With some clever formatting and using <code>include_example</code> a similar table layout can be achieved. Consider this example:</p>
<pre class="brush: ruby">
RSpec.describe 'eating apples' do
def method_under_test(start, eat, left)
start - eat == left
end
RSpec.shared_examples 'eating_apples' do |example|
it "calculates the remaining apple for - #{example}" do
parsed_example = example.split('|')
start, eat, left = parsed_example.reject(&:blank?).map(&:to_i)
result = method_under_test(start, eat, left)
expect(result).to eq(true)
end
end
# start | eat | left
# ______|_____|_____
include_examples 'eating_apples', ' 12 | 5 | 7'
include_examples 'eating_apples', ' 20 | 5 | 15'
end
</pre>
<p>The table is shifted to the right, but it’s clearly visible, and the commented out header provides a different color in my IDE that separates it from the data.</p>
<div style="text-align: center; padding-top: 10px; padding-bottom: 10px;">
<img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/06/include_examples.png" alt="include_examples">
</div>
<p>The <code>shared_example</code> is also a lot simpler, I don’t need to deal with the header information, as it’s not passed to it, examples are being ran line by line.</p>
<p>By making the test description dynamic, the reporter prints out the tested values:</p>
<pre class="brush: shell">
eating apples
calculates the remaining apple for - 12 | 5 | 7
calculates the remaining apple for - 20 | 5 | 15
</pre>
<p>The triggered error is properly reported by both the line number and the example used for that particular use case.</p>
<pre class="brush: shell">
.F
Failures:
1) eating apples calculates the remaining apple for - 20 | 5 | 16
Failure/Error: expect(result).to eq(true)
expected: true
got: false
(compared using ==)
</pre>
<p>I am not a big fan of adjusted white space for positioning equal signs under one another, but am happy to use white space to format test code to better describe the logic it validates.</p>
<p>RSpec does not have scenario outlines, but hopefully, with a <code>shared_example</code> and a bit of clever formatting the same can be accomplished.</p>
Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-70221714081211433342017-05-31T09:22:00.000-05:002017-05-31T11:07:44.856-05:00Bit Shifting for a Shard ID in Ruby<p>As our database grew, we had to take a serious look at how can we could split it up by our clients, as most of them wanted to have their own data separate from the others anyway. A few months ago I found a <a href="https://medium.com/@Pinterest_Engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f">great article</a> from Pinterest, that describes how they <a href="https://en.wikipedia.org/wiki/Shard_(database_architecture)">sharded</a> their MySQL database.</p>
<p>A sharded entity needs a UUID to uniquely identify the record across all shards. Most of the programming languages can generate a UUID easily, however, what was amazing to me, was that Pinterest generated its own unique ids by encapsulating three distinct numbers in one. Wait, what??! Read that article, it’s definitely worth your time.</p>
<p>While Pinterest encapsulated three numbers into one, we only needed two, a <code>client_id</code> and an <code>entity_id</code>. Our <code>client_id</code> would be a much smaller number than our <code>entity_id</code>, we wanted to reserve more bits for the latter.</p>
<p>It turns out, Ruby has many friendlier tools to deal with binary operations. Let's look at them!</p>
<p>What is the binary representation of the integer number 123?</p>
<div style="text-align: center; padding-top: 10px; padding-bottom: 10px;">
<img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/05/123_binary.jpg" alt="123-in-binary" width="500">
</div>
<p>Out of the 7 bits, you'll see that the 3rd one is turned off, all the others are turned on, giving us 64 + 32 + 16 + 8 + 2 +1 = 123. How can we get this binary representation in Ruby? It's super easy, just use the <a href="https://ruby-doc.org/core-2.2.2/Fixnum.html#method-i-to_s">to_s(2)</a> method to do it.</p>
<pre class="brush: ruby">
pry(main)> 123.to_s(2)
=>"1111011"
</pre>
<p>This is the exact same string representation as the one in the image above, where the third bit is turned off and represented with a zero.</p>
<p>I'd like to keep the <code>client_id</code> on the left side, but I'd like to reserve bits on the right side. For the sake of simplicity, I will keep this as a small number. Let's add 5 bits to the right-hand side of these bits by using the bitwise left shift operator.</p>
<pre class="brush: ruby">
pry(main)> (123 << 5).to_s(2)
=> "111101100000"
</pre>
<p>The original number, 123, is still represented on the left side, but 5 "0"s were added to the right-hand side. You get the numeric representation of this by leaving out the <code>to_s(2)</code> call at the end:</p>
<pre class="brush: ruby">
pry(main)> 123 << 5
=> 3936
</pre>
<p>This number can be converted back to binary:</p>
<pre class="brush: ruby">
pry(main)> 3936.to_s(2)
=> "111101100000"
</pre>
<p>On the left side I have the binary representation of 123, but how about those bits on the right side? What are those representing? Right now, those bits are all turned off, they will give you 0 (<code>"00000".to_i(2) => 0</code>). How can I store the number 3 on the right side? The bits should look like this:</p>
<div style="text-align: center; padding-top: 10px; padding-bottom: 10px;">
<img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/05/3_on_right_side.jpg" alt="3-on-right-side" width="200">
</div>
<p>The binary "|" will turn the two rightmost bits on:</p>
<pre class="brush: ruby">
pry(main)> (123 MM 5 | 3).to_s(2)
=> "111101100011"
</pre>
<p>Again, leaving out the <code>to_s(2)</code> will provide the number representation:</p>
<pre class="brush: ruby">
pry(main)> (123 << 5 | 3)
=> 3939
</pre>
<p>The storing part will work, but how can we get our two numbers back from this one combined number? Well, we have to split the bits and convert the binary representation to an integer.</p>
<p>Five bits were used on the right side to store our second number. We need to chop those off to get the number stored on the left side. The bitwise right shift (>>) will do just that:</p>
<pre class="brush: ruby">
pry(main)> (3939 >> 5).to_s(2)
=> "1111011"
</pre>
<p>The string "1111011" is our original 123 in a binary string format. We can convert that to an integer by using the <a href="http://ruby-doc.org/core-2.4.1/String.html#method-i-to_i">to_i(2)</a> String method:</p>
<pre class="brush: ruby">
pry(main)> (3939 >> 5).to_s(2).to_i(2)
=> 123
</pre>
<p>I right shifted the original number, 3939, converted it to a binary string and converted that to an Integer.</p>
<p>There are more efficient ways to do this by using a binary "&" <code>(3939 >> 5) & 0b1111111 => 123</code> with the max value the bits can represent. That's what the Pinterest article had, but I found using the Ruby conversion methods a bit more friendly to those of us who are not dealing with binary data on a daily basis.</p>
<p>We have the number on the left side, but what about the number on the right side? When we convert the number representation (3939) to a binary string, we know that the five characters on the right side will represent the bits of our other number. Ruby String’s <code>last(x)</code> will do just that:</p>
<pre class="brush: ruby">
pry(main)> (3939 >> 0).to_s(2).last(5)
=> "00011"
</pre>
<p>Converting this binary String to Integer should be similar to what we've done before:</p>
<pre class="brush: ruby">
pry(main)> (3939 >> 0).to_s(2).last(5).to_i(2)
=> 3
</pre>
<p>Using the binary "&" with the max number the bits can store will do this conversion in one step: <code>(3939 >> 0) & 0b11111 => 3</code>. As a side note, the binary number can be represented as a hexadecimal value: <code>(3939 >> 0) & 0x1F => 3</code>. This is a lot shorter than a series of ones and zeros.</p>
<p>There is a limit of how large the numbers can be as you have a limit of bits to store those. The max number can be determined by flipping the available bits on. For an 7 bit number it's <code>64 + 32 + 16 + 8 + 4 + 2 + 1 = 127</code> or <code>2**x-1</code>, where x is the number of bits. In our case it is <code>2**7-1 = 127</code>.</p>
<p>We ended up using a 64-bit Integer for our <code>shard_id</code>, which is a BIGINT in MySQL. We store <code>client_id</code> in 22 bits giving us the maximum of <code>2**22-1 = 4_194_304</code> and 40 bits for the <code>entity_id</code> with <code>2**40-1 = 1_099_511_627_775</code> max value. The remaining two bits are "worth in gold, we can expand the number or store a third (albeit small) number in it.</p>
Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-33368704889461797132017-04-25T13:44:00.000-05:002017-04-25T13:44:08.809-05:00Fireside Chat<p>When I saw a retweet from <a href="https://twitter.com/jasonfried">Jason Fried</a> about available tickets to a fireside chat with him at Basecamp, I jumped on it. I figured if I can kill two birds with one stone, - meeting him in person and seeing their offices - it's a no-brainer. Company Culture was the topic of the conversation led by <a href="https://twitter.com/AimeeGroth">Aimee Groth</a>, who visited Chicago to publicize her new book, <a href="https://www.amazon.com/Kingdom-Happiness-Inside-Hsiehs-Zapponian/dp/1501129902">Kingdom of Happiness</a> about Zappos' culture.</p>
<div style="text-align: center; padding-top: 10px; padding-bottom: 10px;">
<img alt="Basecamp HQ" src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/04/basecamp.jpg" width="500"/>
</div>
<p><a href="https://basecamp.com/about/office">Basecamp HQ</a> is as cool as you think it is. Very few desks, a couple of meeting rooms. It reminded me more of a train terminal with its large windows and limited furnishing than a real office. The office is centered around an auditorium, which is an effective PR and educational platform for the company.</p>
<p>I enjoyed looking at the walls covered with postcards from employees all over the world, but I especially liked David's H-1B approval notice from the USCIS from 2005. I laughed out loud when I noticed it, as I had to go through similar hassle myself, but mine is safely guarded with my documents at home.</p>
<p>Basecamp works in six weeks work schedule. Whatever the team can get down in six weeks, they will deliver it. The scope can change, but the six weeks schedule is hard set. This timeframe helps them delivering functionality, and since the company is working remotely, it worked out well for them.</p>
<p>They don't have managers who only manage people or projects, the teams are led by team leads. These team leads are developers as well, shipping code on a daily basis. Jason and his team realized that managers who do not code, grow away from the work. According to him, "professional (full time) managers forget to do the work".<br>
At one point they've tried rotating team leads, but that did not work out, as the continuity was lost. I could see that: "I see this problem, but I won't deal with it, I'll leave it for the next person, who will take over." Basecamp is looking for people who are self-managed, however, Jason emphasized multiple times: "people like to be led". It's important to "rally the folks by clear goals and purpose".</p>
<p>Jason also talked about the Jeff Bezos investment in the company, which meant a small ownership stake in Basecamp. They did not need the money to survive, David and Jason felt having a person like Mr. Bezos is mutually beneficial to both parties. "Who would not like to have Jeff Bezos as an advisor in his or her company?!" They have not talked to Jeff Bezos for a while, but if they wanted to, they could just reach out to his secretary, set up a meeting, and Jeff would fly to Chicago for a meeting or dinner with them.</p>
<p>The best advice from Bezos - and according to Jason, this was worth the entire dividend they have paid for his investment - was: "invest in the things in your business, that won't change". Don't chase the shiny new things, stick to what will not change. For them, it's Basecamp. The company had 4-5 products that they sold a couple of years ago to focus on their main product, which is Basecamp.</p>
<p>Jason went into details why other products were divested (like HighRise, Backpack, Campfire). Maintaining the web, Android and iOS versions of their products resulted in 15 different projects. That led to insufficient focus for each platform for each product with the employees they had at the time. They could - of course - have hired other developers, but they intentionally wanted to stay small. They did not want to get richer, be the next billionaire, they were just as happy with what they had. This sounds strange, almost naive in the era of bloated startups that are bleeding money chasing to be the next Facebook.</p>
<p>I enjoyed the Q&A at the very end. Some interesting questions came up about the startup community in Chicago, about VCs in general. Jason kindly offered to stay as long as everybody's questions were answered. Really a courteous offer, considering it was after 8 pm on a Friday night.</p>
<p>Oh, yes, and one more thing: Basecamp has 130,000 paying customers. It's a remarkable achievement by a company that has never taken VC money, was profitable from the get-go, and created an exciting app in the "not-so-exciting" domain of project management.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-59692453863373880722017-03-28T12:25:00.000-05:002017-03-28T12:26:58.465-05:00Containers<p>As I was exploring how to make Golang even faster on AWS Lambda, I found a project that promised sub-millisecond execution time compared to my (already pretty good) <a href="http://www.adomokos.com/2016/06/using-ruby-with-activerecord-in-aws.html">~60 millisecond</a>. It used Python execution that ran the Go code in-process in contrast to my attempt, where I had to spawn a new process and execute the lambda code there. Very clever, no wonder that solution did not have the 60-millisecond penalty for running that code. However, in order to build the sample code for this AWS Lambda I had to use Docker.</p>
<div style="text-align: center; padding-top: 10px; padding-bottom: 10px;">
<img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/03/docker_logo.png" width="200"/>
</div>
<p>I've heard about Docker years ago, understood what it's used for at a very high level, however, I have never really given it a serious try. I figured it was time. Boy, I was in for some pleasant surprise!</p>
<p>The project <a href="https://github.com/eawsy/aws-lambda-go">AWS Lambda Go from Eawsy</a> used Docker to containerize their build environment on my laptop. What does that mean? Imagine having a build server running on your computer in seconds, where the version of the Go compiler, the Python environment is set by the author of the Dockerfile. I'd use a little build engine that takes in my code, runs its magic and a zip file comes out that I can run on Lambda. What?!</p>
<p>I wrote all these different tutorials about <a href="http://www.adomokos.com/2016/06/using-ruby-with-activerecord-in-aws.html">running MRI Ruby on AWS Lambda</a> or <a href="http://www.adomokos.com/2015/10/clojure-api-with-yesql-migrations-part2.html">interacting with a Postgres DB with Clojure</a> and I had to set up all the prereqs in plain text: "you have to have Postgres running, and Clojure, and MRI Ruby". I provided all the different Makefile scripts to follow the examples. However, with Docker, I'd just provide a Dockerfile that sets up the environment in the future.</p>
<p>I believe containers are big and will be even bigger very soon.</p>
<p>
<p id="highlight">I see more and more applications where the code describes the behavior and the container descriptor describes the environment.</p></br>They live side by side, clearly stating what virtual components the software needs to execute. Engineers can run the software with those containers locally, and the software can be deployed to the cloud with those images pre-built, with tight control over its execution context.
</p>
<p>There are many resources to learn Docker. I started with reading the <a href="https://www.manning.com/books/docker-in-action">Docker in Action</a> book and went further by reading the <a href="https://www.manning.com/books/docker-in-practice">Docker in Practice</a> book.</p>
<p>I created a <a href="https://github.com/adomokos/docker_templates">Docker templates repository</a>, where I collected ideas for different recipes. Do I need a Ruby worker with Redis and Postgres backend? I'll just run <code>docker compose up</code> with this <a href="https://github.com/adomokos/docker_templates/blob/master/ruby/docker-compose.yml"> docker_compose.yml file</a> and I have an environment, where everything from the OS to the version of Redis and Postgres is predefined. If it works on my machine, it will work on yours, too.</p>
<p>There are many things I like about Docker as compared to Vagrant or other virtual machine solutions. The biggest thing for me is the low power Docker containers would need. Vagrant images would reserve 2 of your 4 cores and 8GB memory when Docker will only take from the host as much as it needs. If it's 32MB, that's it, if it's 1GB, it will take that much.</p>
<p>Docker is the future, and you will see more and more code repos with a <code>Dockerfile</code> in it.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-91074211313364770212017-02-08T20:56:00.000-06:002017-02-10T13:28:03.432-06:00Golang<p>The first time I heard about Golang was a few years back, when the great guys at Brad's Deals, our next door office neighbor organized and hosted the local Go meetup there. Then <a href="https://iojs.org/en/">IO.js</a> and Node.js war broke out and TJ Holowaychuck shifted from Node.js to Golang announcing the move in an <a href="https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e52b#.fada6ndrw">open letter</a> to the community.<br>
I did not think much of the language, as its reputation was far from the beauty of a real functional language.</p>
<p>Fast forward a couple of years and I am giving Ruby a serious try on AWS Lambda. <a href="http://www.adomokos.com/2016/06/using-ruby-with-activerecord-in-aws.html">Ruby works there</a>, however, it needs enough memory and 3000 ms (3 seconds) to do anything. We have to invoke some of them millions of times in a month and when we <a href="https://s3.amazonaws.com/lambda-tools/pricing-calculator.html">calculated</a> the cost for it, the bill gets fairly large quickly.</p>
<p>
I created a simple AWS Lambda with Ruby just to print the words "Hello, World!" with 128 MB memory. It took 5339 ms to execute it.
</p>
<div style="text-align: center;"><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/02/ruby_hello_world.png"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/02/ruby_hello_world.png" alt="Ruby Hello World on AWS Lambda" width="500"></a></div>
<p>Then one night I wrote a tiny Go program:</p>
<pre class="brush: golang">
package main
import "fmt"
func main() {
fmt.Println("Hello, World!")
}
</pre>
<p>I cross compiled (since I am working on OSX) with the command <code>GOOS=linux GOARCH=amd64 go build github.com/adomokos/hello</code> to Linux, packaged it up with a Node.JS executor and ran it. I couldn't believe my eyes, it took only 68 ms to get the string "Hello, World!" back. 68 ms! And it was on a 128 MB memory instance. It was beautiful!
</p>
<p>
<div style="text-align: center;"><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/02/go_hello_world.png"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2017/02/go_hello_world.png" alt="Go Hello World on AWS Lambda" width="500"></a></div>
</p>
<p id="highlight">Ruby would need four times the memory and it would still execute ~10 times slower than Go. That was the moment when I got hooked.</p>
<p>Go is a simple language. I am not saying it's easy to learn, it's subjective: it depends on your background, your experience. But it's far from the beauty of Haskell or Clojure. However, the team I am working with would have no trouble switching between Go and Ruby multiple times a day.</p>
<p>What kind of a language today does not have <code>map</code> or <code>reduce</code> functions?! Especially when functions are first-class citizens in the language. It turns out, I can write my own map function if I need to:</p>
<pre class="brush: golang">
package collections
import (
"github.com/stretchr/testify/assert"
"strconv"
"testing"
)
func fmap(f func(int) string, numbers []int) []string {
items := make([]string, len(numbers))
for i, item := range numbers {
items[i] = f(item)
}
return items
}
func TestFMap(t *testing.T) {
numbers := []int{1, 2, 3}
result := fmap(func(item int) string { return strconv.Itoa(item) }, numbers)
assert.Equal(t, []string{"1", "2", "3"}, result)
}
</pre>
<p>Writing <code>map</code> with recursion would be more elegant, but it's not as performant as using a slice with defined length that does not have to grow during the operation.</p>
<h4>History</h4>
<p>Go was created by some very smart people at Google, I wanted to understand their decision to keep a language this pure.<br>
Google has a large amount of code in C and C++, however, those languages are far from modern concepts, like parallel execution and web programming to name a few. Those languages were created in the 60s and 80s, well before the era of multi-core processors and the Internet. Compiling a massive codebase in C++ can easily take hour(s), and while they were waiting for compilation, the idea of a fast compiling, simple, modern language idea was born. Go does not aim to be shiny and nice, no, its designers kept it:</p>
<ul>
<li>to be simple and easy to learn</li>
<li>to compile fast</li>
<li>to run fast</li>
<li>to make parallel processing easy</li>
</ul>
<p>Google hires massive number of fresh CS graduates each year with some C++ and Java programming experience, these engineers can feel right at home with Go, where the syntax and concept is similar to those languages.</p>
<h4>Tooling</h4>
<p>Go comes with many built-in tools, like code formatting and benchmarking to name the few. In fact I set up <a href="https://github.com/fatih/vim-go">Vim Go</a> that leverages many of those tools for me. I can run, test code with only a couple of keystrokes.</p>
<p>Let's see how performant the procedure I wrote above is. But before I do that I'll introduce another function where the slice's length is not pre-determined at the beginning of the operation, this way it has to auto-scale internally.</p>
<pre class="brush: golang">
func fmapAutoScale(f func(int) string, numbers []int) []string {
// Initialize a slice with default length, it will auto-scale
var items []string
for _, item := range numbers {
items = append(items, f(item))
}
return items
}
</pre>
<p>The function is doing the same as <code>fmap</code>, similar test should verify the logic.</p>
<p>I added two benchmark tests to cover these functions:</p>
<pre class="brush: golang">
// Run benchmark with this command
// go test -v fmap_test.go -run="none" -benchtime="3s" -bench="BenchmarkFmap"
// -benchmem
func BenchmarkFmap(b *testing.B) {
b.ResetTimer()
numbers := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
for i := 0; i < b.N; i++ {
fmap(func(item int) string { return strconv.Itoa(item) }, numbers)
}
}
func BenchmarkFmapAutoScale(b *testing.B) {
b.ResetTimer()
numbers := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
for i := 0; i < b.N; i++ {
fmapAutoScale(func(item int) string { return strconv.Itoa(item) },
numbers)
}
}
</pre>
<p>When I ran the benchmark tests, this is the result I received:</p>
<pre class="brush: golang">
% go test -v fmap_test.go -run="none" -benchtime="3s" -bench="BenchmarkFmap"
-benchmem
BenchmarkFmap-4 ‡ 10000000 | 485 ns/op | 172 B/op | 11 allocs/op
BenchmarkFmapAS-4 ‡ 5000000 | 851 ns/op | 508 B/op | 15 allocs/op
PASS
ok command-line-arguments 10.476s
</pre>
<p>The first function, where I set the slice size to the exact size is more performant than the second one, where I just initialize the slice and let it autoscale. The <code>ns/op</code> displays the execution length per operation in nanoseconds. The <code>B/op</code> output describes the bytes it uses per operation. The last column describes how many memory allocations it uses per operation. The difference is insignificant, but you can see how this can become very useful as you try writing performant code.</p>
<h4>Popularity</h4>
<p>Go is getting popular. In fact, very popular. It was TIOBE's "<a href="http://www.tiobe.com/tiobe-index/">Language of the Year</a>" gaining 2.16% in one year. I am sure you'll be seeing articles about Go more and more. Check it out if you haven't done it yet, as the chance of finding a project or job that uses Go is increasing every day.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-82587296093324326682016-11-13T18:42:00.000-06:002016-11-14T13:33:53.317-06:00Recursion Done Right - Haskell Influenced Recursion in Ruby<p>Learning Haskell has influenced the way I think about code. I wrote about <a href="http://www.adomokos.com/2016/05/currying-in-haskell-clojure-ruby-and.html">currying</a> before in various languages, but Haskell taught me a bit about how to do recursion properly.</p>
<p>Although fast paced, I really like the examples in the book <a href="http://learnyouahaskell.com/">Learn you a Little Haskell for Great Good</a>. As <a href="http://learnyouahaskell.com/recursion#maximum-awesome">one chapter</a> talks about recursion and higher order functions, I was amazed by the simplicity of the code that lets you do basic list operations.</p>
<p>Here is how one could find the maximum of a list:</p>
<pre class="brush: haskell">
maximum' :: (Ord a) => [a] -> a
maximum' [] = error "maximum of an empty list"
maximum' (x:xs) = max x (maximum' xs)
</pre>
<p>There is already a <code>maximum</code> function in Haskell's core library, this example just shows you what you need to do to implement it yourself.</p>
<p>I am not going into details about the type declaration, but there are a couple of points I'd like to talk about.
The pattern matching in the second line checks for the case, where the collection is an empty array. When that happens, an exception is thrown.
The last line does pattern matching as well, it grabs the head and the tail of the list and saves it into the x and xs variables. Then it uses the <code>max</code> functions to figure out which number is greater: x or the recurred result of maximum' with the tail of the list.
This is a prime example of declarative code, its simplicity is striking and the fact that I don’t have to know how max works with the recursion makes it a joy to read.</p>
<p>Let’s look at another example. Here is how you could implement <code>map</code> in Haskell yourself:</p>
<pre class="brush: haskell">
map' :: (a -> b) -> [a] -> [b]
map' f [] = []
map' f (x:xs) = f x : map' f xs
</pre>
<p>Similarly, the edge-case is handled first. The last line has the logic, function <code>f</code> is applied to the head of the list, the result is concatenated with the recurred result of <code>map'</code> of f function and the tail of the list.</p>
<p>All right, let’s see how we could express this logic in Ruby.</p>
<p>Here is my first attempt:</p>
<pre class="brush: ruby">
module Collections
def self.maximum(collection)
head = collection.first
tail = collection[1..-1]
return 0 unless tail
max head, (maximum tail)
end
def self.max(a, b)
a > b ? a : b
end
private_class_method :max
end
RSpec.describe 'Recursion done right' do
context 'maximum' do
it 'returns an empty array as max of empty list' do
expect(Collections.maximum([])).to eq(0)
end
it 'returns the maximum of a list' do
expect(Collections.maximum([1,3,2])).to eq(3)
end
end
end
</pre>
<p>I did not find a <code>max</code> method in Ruby, I added that as a private class method. This is still pretty easy to read, but a bit more verbose than what I'd like it to be. I wanted to find the <code>(x:xs)</code> head-tail (<a href="https://en.wikipedia.org/wiki/CAR_and_CDR">car-cdr</a> for you LiSP folks) equivalent in Ruby, I knew that will be key to make it a more succinct solution. This is it: <code>(head, *tail) = collection</code>. I also had to change the guard to quit from the recursion to look for an empty array, as the splat operator will provide that.</p>
<p>Here is my revised solution:</p>
<pre class="brush: ruby">
module Collections
def self.maximum(collection)
(head, *tail) = collection
return 0 if tail.empty?
max head, (maximum tail)
end
...
end
</pre>
<p>This is better, but the destructuring can take place in the argument:</p>
<pre class="brush: ruby">
module Collections
def self.maximum((head, *tail))
return 0 if tail.empty?
max head, (maximum tail)
end
...
end
</pre>
<p>This is pretty darn close to the solution in Haskell.<br>
Now let’s look at the map function.</p>
<p>These specs describe the behavior:</p>
<pre class="brush: ruby">
context 'map' do
it 'mapping [] with (*3) gives []' do
expect(Collections.map(->(x){ x*3 }, [])).to be_empty
end
it 'mapping [1,2,3] with (*3) gives [1,6,9]' do
expect(Collections.map(->(x){ x*3 }, [1,2,3])).to eq([3,6,9])
end
end
</pre>
<p>My implementation of <code>map</code> takes a lambda with one argument, which multiplies that one argument by three, and the second argument is the collection of items the map function will operate on.</p>
<p>This is my implementation for it:</p>
<pre class="brush: ruby">
module Collections
def self.map(f, (head, *tail))
return [] unless head
[f.(head)] + map(f, tail)
end
...
end
</pre>
<p>The key to make it concise is the destructuring the collection argument into head and tail. The guard statement makes sure the recursion will quit once there is no item in the head. The bulk of the logic is the last line of the method: the lambda is applied to the head, it's converted into an array and that value is concatenated with the result of the recurred result of the lambda and the rest of the collection.</p>
<p>In our case, the following calculation takes place:</p>
<pre class="brush: shell">
map (*3) [1,2,3]
[(3*1)] + map (*3) [2,3]
[(3*1)] + [(3*2)] + map (*3) [3]
[(3*1)] + [(3*2)] + [(3*3)]
[3] + [6] + [9]
[3,6,9]
</pre>
<p>Haskell takes pride in how easily it implements the <a href="https://en.wikipedia.org/wiki/Quicksort">quicksort algorithm</a>. Let’s see how it’s done there:</p>
<pre class="brush: haskell">
quicksort :: (Ord a) => [a] -> [a]
quicksort [] = []
quicksort (x:xs) =
let smallerSorted = quicksort [a | a <- xs, a <= x]
biggerSorted = quicksort [a | a <- xs, a > x]
in smallerSorted ++ [x] ++ biggerSorted
</pre>
<p>I don’t blame you if this seems to be a bit more cryptic than you wanted to be. It takes a little practice to read what is really going on here. I'll explain it, as it will help our own Ruby implementation. The first line is the type declaration, ignore that for now. The second line is the guard, sorting an empty array will give you back an empty array. The meat of the logic begins on the third line. The collection argument is destructured into head and tail, just like I've been doing in the examples above. Based on the head value, we are filtering the elements into smaller-equal, and bigger parts. We do all this recursively until the list is exhausted. Right before the result is returned, the three items, the smaller sorted, the head value and the bigger sorted elements are combined into one collection.</p>
<p>Let’s see how this is done in Ruby. Here are the specs I prepared to prove the logic:</p>
<pre class="brush: ruby">
context 'quicksort' do
it 'returns an empty list for empty list' do
expect(Collections.quicksort([])).to eq([])
end
it 'sorts a list of items' do
expect(Collections.quicksort([2,5,3])).to eq([2,3,5])
end
end
</pre>
<p>Here is how I'd like the code to be:</p>
<pre class="brush: ruby">
def self.quicksort((head, *tail))
return [] unless head
smaller_sorted = quicksort(Collections.filter(->(x) { x <= head }, tail))
bigger_sorted = quicksort(Collections.filter(->(x) { x > head }, tail))
smaller_sorted + [head] + bigger_sorted
end
</pre>
<p>This logic is very close to the Haskell example, but unfortunately, I don't have the filter function just yet. (Ruby standard library offers the <code>select</code> method on enumerables, but let's keep these examples free from all that.) <code>filter</code> takes a lambda as its predicate function, and a collection it needs to operate on.
This spec proves out our logic:</p>
<pre class="brush: ruby">
context 'filter' do
specify 'filter (>2) [] returns an empty list' do
expect(Collections.filter(->(x){ x > 2 }, [])).to be_empty
end
specify 'filter (>2) [1,3,5] returns [3,5]' do
expect(Collections.filter(->(x){ x > 2 }, [1,3,5])).to eq([3,5])
end
end
</pre>
<p>And the implementation is similar what you've seen before:</p>
<pre class="brush: ruby">
def self.filter(f, (head, *tail))
return [] unless head
if f.(head)
[head] + filter(f, tail)
else
filter(f, tail)
end
end
</pre>
<p>And now, when you run the entire spec, the quicksort implementation just magically works.</p>
<div style="text-align: center;"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/11/specs_executed.png" width="480" alt="specs executed"></div>
<p>Studying Haskell taught me a few things about recursion. The head and tail concept is essential to make the code simple and neat. Without that it would have been a lot more noisier. Whenever I used recursion before, I always felt I needed an accumulator. I wanted something I could jump to and investigate when something went wrong. I would have written the filter function like this before:</p>
<pre class="brush: ruby">
def self.filter(f, (head, *tail), accumulator=[])
return accumulator unless head
accumulator << head if f.(head)
filter(f, tail, accumulator)
end
</pre>
<p>Although this works, adding the accumulator with a default argument to the list just makes this code a lot noisier, but I do like not having conditional branches in it, it's just easier to reason about this code.</p>
<p>
You can review the examples in <a href="https://gist.github.com/adomokos/cc326bf27b0529d9386813c8af66e59c">this gist</a>.
</p>
<p>Based on what you read here, try implementing <code>replicate</code>, <code>take</code>, <code>reverse</code>, <code>repeat</code> and <code>zip</code> functions yourself. In case you need directions, check out this <a href="https://gist.github.com/adomokos/3182e9914ab29b5a36c444a16ae7ecd1">gist</a> to see how I did it.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-85693147915565303202016-10-21T21:21:00.000-05:002016-10-21T21:21:31.101-05:00Vim Katas<p>It was about 5 years ago when I watched Jim Weirich solving the <a href="https://vimeo.com/33841375">Roman Numeral kata</a> at an SCNA conference in Chicago. I was amazed by how he mastered his editor of choice: Emacs. His fingers were flying on the keyboard and he wrangled the code I have rarely seen anybody before that.</p>
<div style="text-align: center;"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/10/vim_logo.png" width="100" alt="aws-lambda"></div>
<p>I started using Vim in 2008 or 2009, but I never really invested the time to be good at it. I read the man pages, I went through Vim tutor, but I never really picked up or started using most of the advanced features.</p>
<p>I remember how great I felt when I reset my Caps Lock key to function as Ctrl key. The power of hitting <code><Ctrl+c></code> with my pinky and index finger just to trigger <code><Esc></code> without reaching up to the top left on the keyboard made me feel I've just found kryptonite.</p>
<p>I've had the book <a href="http://pragprog.com/practical_vim">Practical Vim</a> for some time, but I never really practiced the examples in it. I looked at them here and there, tried them out, but I've never made a habit of practicing those daily. Then one day I got sick and tired of my inabilities, I started a new markdown document where I jotted down the <a href="https://github.com/adomokos/Vim-Katas/blob/master/24_editing_tabular_data_with_visual_block_mode.md">first exercise</a> and the project of <a href="https://github.com/adomokos/Vim-Katas">Vim Katas</a> was born. Every time I commuted to work, I started with the first one and practiced all of them. Once I got to the end of it, I read the book further and added new exercises to it.</p>
<p>I might have covered 60% of the book by now, but when I bump into a repeatable task and I leverage a macro for it, it always puts a smile on my face.</p>
<p>Using Vim reminds me of learning to play a musical instrument. It takes time and effort to be good at it, and the keystrokes have to be at your fingertips. Once you slow down and think about it, the effectiveness of the text-based editor fades away, you would be more efficient by using visual editor instead (like Sublime Text, IntelliJ or Visual Studio).</p>
<p>Vim (and I am sure many other tools) has this huge wall that people have to climb first to appreciate it. Clone <a href="https://github.com/adomokos/Vim-Katas">that repo</a>, open the first markdown file in Vim, and start practicing!</p>
Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-1225689623544346872016-06-06T20:05:00.001-05:002016-06-09T09:55:55.314-05:00Using Ruby with ActiveRecord in AWS Lambda<p>In my <a href="http://www.adomokos.com/2016/06/using-ruby-in-aws-lambda.html" target="_blank">previous blog post</a> I described how you can run MRI Ruby on AWS Lambda. In this article I'll guide you through adding gems to the project: first <a href="https://github.com/stympy/faker" target="_blank">faker</a>, and then the <a href="https://github.com/brianmario/mysql2" target="_blank">mysql2</a> gem with <a href="https://github.com/rails/rails/tree/master/activerecord" target="_blank">activerecord</a>, and finally we will have the Ruby code talk to an RDS instance, all this through an AWS Lambda.</p>
<p>I recorded all my changes in <a href="https://github.com/adomokos/aws-lambda-ruby" target="_blank">this project</a>, feel free to jump in where you want, you'll find commit points after each section.</p>
<h4>1. Add faker to the project</h4>
<p>You can pick up the changes from the previous blog post <a href="https://github.com/adomokos/aws-lambda-ruby/commit/7bf6c4e5d6f745d636dbdc6737db7f23a4371085" target="_blank">here</a>. Let's start by editing our Ruby app, add a Gemfile to it in the <code>hello_ruby</code> directory:</p>
<pre class="brush: ruby">
source 'https://rubygems.org'
gem 'faker'
</pre>
<p>Run <code>BUNDLE_IGNORE_CONFIG=1 bundle install --path vendor</code> in that directory. The <code>--path vendor</code> argument is important, as we have to package all the files in the <code>vendor</code> directory. Make sure the <code>BUNDLED WITH</code> part of your Gemfile.lock is not there as that can cause you some pain when you deploy your code to AWS Lambda.</p>
<p>Edit the <code>lib/hello.rb</code> file like this:</p>
<pre class="brush: ruby">
#!/usr/bin/env ruby
require 'faker'
puts "Hello - '#{Faker::Name.name}' from Ruby!"
</pre>
<p>We required the faker gem and used it to generate a fake name. Run the app in the terminal with <code>bundle exec ruby lib/hello.rb</code> command.</p>
<pre class="brush: shell">
Hello - 'Jamar Gutmann II' from Ruby!
</pre>
<p>You will get a different name between the single quotes, but that's the point, faker generates a random name for us.</p>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/a0ca3becd83e3a2c6c87f627114f20dfdcbffd5f" target="_blank">Commit point</a></p>
<h4>2. Use faker with Traveling Ruby</h4>
<p>The <code>run</code> target in the Makefile will have to copy all vendorized gems, plus it needs to configure the app to run with the correct bundler settings. This step is heavily influenced by how <a href="http://phusion.github.io/traveling-ruby/" target="_blank">Traveling Ruby</a> packages gems for deployment, please review <a href="https://github.com/phusion/traveling-ruby/blob/master/TUTORIAL-2.md" target="_blank">their tutorial</a> as reference.</p>
<p>Add a <code>bundler_config</code> template file to the <code>resources</code> directory with this content:</p>
<pre class="brush: shell">
BUNDLE_PATH: .
BUNDLE_WITHOUT: development:test
BUNDLE_DISABLE_SHARED_GEMS: '1'
</pre>
<p>Change the <code>resources/wrapper.sh</code> file to set the Gemfile’s location:</p>
<pre class="brush: shell">
#!/bin/bash
set -e
# Figure out where this script is located.
SELFDIR="`dirname \"$0\"`"
SELFDIR="`cd \"$SELFDIR\" && pwd`"
# Tell Bundler where the Gemfile and gems are.
export BUNDLE_GEMFILE="$SELFDIR/lib/vendor/Gemfile"
unset BUNDLE_IGNORE_CONFIG
# Run the actual app using the bundled Ruby interpreter, with Bundler activated.
exec "$SELFDIR/lib/ruby/bin/ruby" -rbundler/setup "$SELFDIR/lib/app/hello.rb"
</pre>
<p>Modify the Makefile's <code>run</code> target with the following changes:</p>
<pre class="brush: shell; highlight: [11,12,13,14,15,16]">
...
run: ## Runs the code locally
@echo 'Run the app locally'
@echo '-------------------'
@rm -fr $(OSXDIR)
@mkdir -p $(OSXDIR)/lib/ruby
@tar -xzf resources/traveling-ruby-20150715-2.2.2-osx.tar.gz -C $(OSXDIR)/lib/ruby
@mkdir $(OSXDIR)/lib/app
@cp hello_ruby/lib/hello.rb $(OSXDIR)/lib/app/hello.rb
@cp -pR hello_ruby/vendor $(OSXDIR)/lib/
@rm -f $(OSXDIR)/lib/vendor/*/*/cache/*
@mkdir -p $(OSXDIR)/lib/vendor/.bundle
@cp resources/bundler-config $(OSXDIR)/lib/vendor/.bundle/config
@cp hello_ruby/Gemfile $(OSXDIR)/lib/vendor/
@cp hello_ruby/Gemfile.lock $(OSXDIR)/lib/vendor/
@cp resources/wrapper.sh $(OSXDIR)/hello
@chmod +x $(OSXDIR)/hello
@cd $(OSXDIR) && ./hello
...
</pre>
<p>Run the target with <code>make run</code> and you should see something similar to this in the terminal:</p>
<pre class="brush: shell">
$: make run
Run the app locally
-------------------
Hello - 'Kelly Huel' from Ruby!
</pre>
<p>We've just run the app with Traveling Ruby's Ruby interpreter, and we used the <code>faker</code> gem's functionality as well!</p>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/3a3cfe89c5a65527be141256c5ab85700d1114ae" target="_blank">Commit point</a></p>
<h4>3. Deploy the app with faker to AWS Lambda</h4>
<p>In order to run your app in AWS Lambda, you only need to change the <code>package</code> target in your Makefile, everything else, the delete, create, invoke targets should remain the same. Change the file like this:</p>
<pre class="brush: shell; highlight: [12,13,14,15,16,17]">
...
package: ## Package the code for AWS Lambda
@echo 'Package the app for deploy'
@echo '--------------------------'
@rm -fr $(LAMBDADIR)
@rm -fr deploy
@mkdir -p $(LAMBDADIR)/lib/ruby
@tar -xzf resources/traveling-ruby-20150715-2.2.2-linux-x86_64.tar.gz -C $(LAMBDADIR)/lib/ruby
@mkdir $(LAMBDADIR)/lib/app
@cp hello_ruby/lib/hello.rb $(LAMBDADIR)/lib/app/hello.rb
@cp -pR hello_ruby/vendor $(LAMBDADIR)/lib/
@rm -f $(LAMBDADIR)/lib/vendor/*/*/cache/*
@mkdir -p $(LAMBDADIR)/lib/vendor/.bundle
@cp resources/bundler-config $(LAMBDADIR)/lib/vendor/.bundle/config
@cp hello_ruby/Gemfile $(LAMBDADIR)/lib/vendor/
@cp hello_ruby/Gemfile.lock $(LAMBDADIR)/lib/vendor/
@cp resources/wrapper.sh $(LAMBDADIR)/hello
@chmod +x $(LAMBDADIR)/hello
@cp resources/index.js $(LAMBDADIR)/
@cd $(LAMBDADIR) && zip -r hello_ruby.zip hello index.js lib/
@mkdir deploy
cd $(LAMBDADIR) && mv hello_ruby.zip ../deploy/
@echo '... Done.
...
</pre>
<p>The added rows are very similar to the ones we had to add to run the app locally with Traveling Ruby. Delete the lambda function and recreate it by using the Makefile. When you invoke it, your should see something like this:</p>
<pre class="brush: shell; highlight: [4]">
START RequestId: 3f6ae8f5-23c1-11e6-9acc-0f50ffa39e9b Version: $LATEST
2016-05-27T04:12:41.473Z
3f6ae8f5-23c1-11e6-9acc-0f50ffa39e9b
Hello - 'Mrs. Lelah Bradtke' from Ruby!
END RequestId: 3f6ae8f5-23c1-11e6-9acc-0f50ffa39e9b
REPORT RequestId: 3f6ae8f5-23c1-11e6-9acc-0f50ffa39e9b
Duration: 3425.01 ms
Billed Duration: 3500 ms
Memory Size: 512 MB
Max Memory Used: 65 MB
</pre>
<p>The <code>Hello - 'xyz' from Ruby!</code> string contains the Faker gem generated name. You can also invoke the Lambda function through the AWS Management Console, you should see something similar to this in the <code>Log output</code> section:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/faker_with_aws_lambda.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/faker_with_aws_lambda.jpg" width="550" alt="faker-with-aws-lambda"></a></p>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/8fa581ded4007f3ae2b5a1ace2da79991f07b75e" target="_blank">Commit point</a></p>
<h4>4. Publish a newer version to AWS Lambda</h4>
<p>Dropping and recreating the Lambda function works, but it's not the most effective solution. AWS allows you to update your function which we'll do with this new target in the Makefile:</p>
<pre class="brush: shell">
...
publish: package ## Deploys the latest version to AWS
aws lambda update-function-code \
--function-name HelloFromRuby \
--zip-file fileb://./deploy/hello_ruby.zip
...
</pre>
<p>This target will let you update the function code. It also calls the <code>package</code> target to make sure your latest changes will be deployed to AWS.</p>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/1edeaca0ad85806a20947ff1df688e660f7f447e" target="_blank">Commit point</a></p>
<h4>5. Create a new RDS database with one table</h4>
<p>Add this script to your Makefile, it will create a minimal RDS instance for you, you can drop that instance, connect to the DB and drop/create the database with some seed data in it.</p>
<pre class="brush: shell">
DBPASSWD=Kew2401Sd
DBNAME=awslambdaruby
...
create-rds-instance: ## Creates an RDS MySQL DB instance
aws rds create-db-instance \
--db-instance-identifier MyInstance01 \
--db-instance-class db.t1.micro \
--engine mysql \
--allocated-storage 10 \
--master-username master \
--master-user-password $(DBPASSWD)
delete-rds-instance: ## Deletes an RDS MySQL DB instance
aws rds delete-db-instance \
--db-instance-identifier MyInstance01 \
--skip-final-snapshot
db-connect: ## Connects to the RDS instance
mysql --user=master --password=$(DBPASSWD) --host myinstance01.cgic5q3lz0bb.us-east-1.rds.amazonaws.com
create-db: ## Creates a DB with a table and records
@echo "Dropping and creating database"
@echo "-------------------------------"
@mysql -u master --password='$(DBPASSWD)' --host myinstance01.cgic5q3lz0bb.us-east-1.rds.amazonaws.com -e "DROP DATABASE IF EXISTS $(DBNAME)" > /dev/null 2>&1
@mysql -u master --password='$(DBPASSWD)' --host myinstance01.cgic5q3lz0bb.us-east-1.rds.amazonaws.com -e "CREATE DATABASE $(DBNAME)" > /dev/null 2>&1
@mysql -u master --password='$(DBPASSWD)' --host myinstance01.cgic5q3lz0bb.us-east-1.rds.amazonaws.com $(DBNAME) < resources/schema.sql > /dev/null 2>&1
@mysql -u master --password='$(DBPASSWD)' --host myinstance01.cgic5q3lz0bb.us-east-1.rds.amazonaws.com $(DBNAME) < resources/seed.sql > /dev/null 2>&1
@echo "... Done"
...
</pre>
<p>Create the RDS instance first, AWS will need some time to initialize it. Allow incoming connections to it by adjusting the "Inbound" traffic through your own IP under your Security Group:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/adjust_security_group.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/adjust_security_group.jpg" width="550" alt="adjust-security-group"></a></p>
<p>You can connect to the RDS instance through the <code>mysql</code> console using the <code>db-connect</code> target. You'll need to adjust the hostname to yours. Once that works out, use the <code>create-db</code> target to create a DB with a table and add two records to it. If all goes well, this is what you should see when you query the <code>users</code> table in the MySQL console:</p>
<pre class="brush: shell">
mysql> SELECT * FROM users;
+----+--------+------------------+-----------+----------+
| id | login | email | firstname | lastname |
+----+--------+------------------+-----------+----------+
| 1 | jsmith | jsmith@gmail.com | John | Smith |
| 2 | bjones | bjones@gmail.com | Bob | Jones |
+----+--------+------------------+-----------+----------+
2 rows in set (0.04 sec)
</pre>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/435c76738689b24034446578d2707d5304544a89" target="_blank">Commit point</a></p>
<h4>6. Connect to MySQL with Rails' ActiveRecord</h4>
<p>Add the <code>mysql2</code> and <code>active_record</code> gems to the Ruby app's Gemfile:</p>
<pre class="brush: ruby">
gem 'activerecord'
gem 'mysql2', '0.3.18'
</pre>
<p>We need to use the 0.3.18 version of the mysql2 gem, as that <a href="https://traveling-ruby.s3-us-west-2.amazonaws.com/list.html" target="_blank">comes packaged</a> with Traveling Ruby. Run <code>bundle install</code> to get the new gems via Bundler.</p>
<p>Modify the <code>lib/hello.rb</code> file to have this:</p>
<pre class="brush: ruby; highlight: [8]">
#!/usr/bin/env ruby
require 'faker'
require 'active_record'
ActiveRecord::Base.establish_connection(
:adapter => "mysql2",
:host => "myinstance01.cgic5q3lz0bb.us-east-1.rds.amazonaws.com", # use your instance name
:username => "master",
:password => "Kew2401Sd",
:database => "awslambdaruby"
)
class User < ActiveRecord::Base
end
puts "Number of users: #{User.count}"
puts "First user: #{User.first.firstname} #{User.first.lastname}"
puts "Hello - '#{Faker::Name.name}' from Ruby!"
</pre>
<p>You need to adjust the <code>:host</code> value to your RDS instance host name as mine won't work for you. You'll know that everything is set up properly when you see this in the terminal:</p>
<pre class="brush: shell">
$: bundle exec ruby lib/hello.rb
Number of users: 2
First user: John Smith
Hello - 'Miss Darrick Powlowski' from Ruby!
</pre>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/6c74f4313f9c8a34e83dfc64c572ddc694f62789" target="_blank">Commit point</a></p>
<h4>7. Use Traveling Ruby’s packaged mysql gem</h4>
<p>You need to download the Traveling Ruby packaged <a href="http://d6r77u77i8pq3.cloudfront.net/releases/traveling-ruby-gems-20150715-2.2.2-linux-x86_64/mysql2-0.3.18.tar.gz" target="_blank">mysql2 gem</a> from their S3 bucket. Let’s put it into our <code>resources</code> directory.</p>
<p>Modify the <code>package</code> target like this:</p>
<pre class="brush: shell; highlight: [13,14]">
...
package: ## Packages the code for AWS Lambda
@echo 'Package the app for deploy'
@echo '--------------------------'
@rm -fr $(LAMBDADIR)
@rm -fr deploy
@mkdir -p $(LAMBDADIR)/lib/ruby
@tar -xzf resources/traveling-ruby-20150715-2.2.2-linux-x86_64.tar.gz -C $(LAMBDADIR)/lib/ruby
@mkdir $(LAMBDADIR)/lib/app
@cp hello_ruby/lib/hello.rb $(LAMBDADIR)/lib/app/hello.rb
@cp -pR hello_ruby/vendor $(LAMBDADIR)/lib/
@rm -fr $(LAMBDADIR)/lib/vendor/ruby/2.2.0/extensions
@tar -xzf resources/mysql2-0.3.18-linux.tar.gz -C $(LAMBDADIR)/lib/vendor/ruby/
@rm -f $(LAMBDADIR)/lib/vendor/*/*/cache/*
@mkdir -p $(LAMBDADIR)/lib/vendor/.bundle
@cp resources/bundler-config $(LAMBDADIR)/lib/vendor/.bundle/config
@cp hello_ruby/Gemfile $(LAMBDADIR)/lib/vendor/
@cp hello_ruby/Gemfile.lock $(LAMBDADIR)/lib/vendor/
@cp resources/wrapper.sh $(LAMBDADIR)/hello
@chmod +x $(LAMBDADIR)/hello
@cp resources/index.js $(LAMBDADIR)/
@cd $(LAMBDADIR) && zip -r hello_ruby.zip hello index.js lib/ > /dev/null
@mkdir deploy
@cd $(LAMBDADIR) && mv hello_ruby.zip ../deploy/
@echo '... Done.'
...
</pre>
<p>We need to replace the content of the <code>2.0.0/extensions</code> directory with the Traveling Ruby's Linux version, as the one copied there is OSX specific.</p>
<p>AWS Lambda has an IP address other than your IP. In order to make it easy for you now, (and don't do this anywhere else), I'd suggest making your AWS Instance available without IP restriction. Do this only temporarily, to test things out, remove this Inbound rule once you've seen your Lamba working. You can specify the VPC your Lambda has access to, but the topic of AWS Lambda security would need another blog post just in itself.</p>
<p>This is how I opened up my RDS instance for any IP out there:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/connect_anywhere.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/connect_anywhere.jpg" width="550" alt="connect-anywhere"></a></p>
<p>If everything is configured properly, you should see something like this in your terminal when you call the Lambda function with the <code>make invoke</code> command:</p>
<pre class="brush: shell; highlight: [13,14,15]">
% make invoke
rm -fr tmp && mkdir tmp
aws lambda invoke \
--invocation-type RequestResponse \
--function-name HelloFromRuby \
--log-type Tail \
--region us-east-1 \
--payload '{"name":"John Adam Smith"}' \
tmp/outfile.txt \
| jq -r '.LogResult' | base64 -D
START RequestId: 8444ede9-26d8-11e6-954c-fbf57aab89fb Version: $LATEST
2016-05-31T02:36:50.587Z 8444ede9-26d8-11e6-954c-fbf57aab89fb
Number of users: 2
First user: John Smith
Hello - 'Jeanne Hansen' from Ruby!
END RequestId: 8444ede9-26d8-11e6-954c-fbf57aab89fb
REPORT RequestId: 8444ede9-26d8-11e6-954c-fbf57aab89fb
Duration: 5072.62 ms
Billed Duration: 5100 ms
Memory Size: 512 MB
Max Memory Used: 53 MB
</pre>
<p>Sweet! The Ruby code in this AWS Lambda function reports back 2 users and correctly displays the first record.</p>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/2f307f68c3d06a23ff9024c303656ec6d6144a0f" target="_blank">Commit point</a></p>
<p>Being able to use MRI Ruby with gems opens up a ton possibilities for us (and I hope for you as well). AWS Lambdas are neat little workers that can scale up and down very well. It's much easier to launch a 1000 AWS Lambdas at the same time than running Ruby processes with <a href="https://github.com/resque/resque" target="_blank">resque</a> or <a href="https://github.com/mperham/sidekiq" target="_blank">sidekiq</a> on worker boxes.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-14035121588039239662016-06-03T09:08:00.000-05:002016-06-06T20:30:34.604-05:00Using Ruby in AWS Lambda<p>It was May 2015 at the AWS Summit in Chicago, where I first heard about <a href="https://aws.amazon.com/lambda/" target="_blank">AWS Lambda</a>. The company I worked for used Linode at that time, I had no chance of using it, but I still found the serverless concept fascinating.</p>
<div style="text-align: center;"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/aws_lambda.png" width="100" alt="aws-lambda"></div>
<p>The bulk of my work at my <a href="https://www.kennasecurity.com/" target="_blank">current gig</a> is about transforming data: we pull it from an API, we need to transform and load it into our own data store. Sure the worker boxes can do the job, but maintaining a series of these instances takes effort. AWS Lambda would be the perfect solution for us, but Amazon does not support Ruby natively, which is most of our business logic is written in.</p>
<p>AWS, as of this writing, offers Lambda for three main platforms: Java, Node.JS, and Python. I played around running Clojure on it, which worked as the code is compiled into a jar file, but our current app - due to its monolithic nature - can’t support any other languages just yet.</p>
<p>Amazon claims you can run your language of choice on AWS Lambda, Ruby included, but I have not found a comprehensive guide that would describe how. Once you can package up your app to run as an executable, you can run it. I found this <a href="https://medium.com/@gigq/using-swift-in-aws-lambda-6e2a67a27e03#.gtg1u3lve" target="_blank">blog post</a> that describes how Swift code can be bundled, deployed and invoked on AWS Lambda. It was clear to me that this solution would work, I only had to package Ruby with its own interpreter to accomplish the same. I looked for tools that can do this and found <a href="http://phusion.github.io/traveling-ruby/" target="_blank">Traveling Ruby</a>. You can package your code and run it as an executable on the user’s computer, no local Ruby installation is needed. I wanted to try it locally first, thinking if it works there (on OSX), it should work on AWS Lambda as well.</p>
<p>
This blog post is a step-by-step tutorial to run MRI Ruby on AWS Lambda. You can follow along with the <a href="https://github.com/adomokos/aws-lambda-ruby" target="_blank">accompanying project</a>, I listed commit points at the end of each section.</p>
<p>This tutorial assumes you are familiar with AWS, you have access to the AWS Management Console and you have the <a href="https://aws.amazon.com/cli/" target="_blank">AWS Command Line Interface</a> configured to interact with your services via the terminal.<br/>
You'll need the same version of Ruby as the one Traveling Ruby offers. The latest there is Ruby 2.2.2, I'd recommend installing that through Rbenv or RVM.</p>
<h4>1. Setting up the project</h4>
<p>I named the project <code>aws-lambda-ruby</code> and created a directory structure like this:</p>
<pre class="brush: shell">
- aws-lambda-ruby
|- hello_ruby
|- lib
|- hello.rb
</pre>
<p>I put this code in the <code>hello.rb</code> file:</p>
<pre class="brush: ruby">
puts 'Hello from Ruby!'
</pre>
<p>I made sure my Ruby version in the project is 2.2.2 by setting it with Rbenv.</p>
<pre class="brush: shell">
$: cd hello_ruby && ruby lib/hello.rb
Hello from Ruby!
</pre>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/c1d1970023ccf376c718aa1516b356df0a6c0d16" target="_blank">Commit point</a></p>
<h4>2. Execute the Ruby Code with Traveling Ruby</h4>
<p>Create a directory under the project root directory with the name <code>resources</code>. Your directory structure should look like this:</p>
<pre class="brush: shell">
- aws-lambda-ruby
|- hello_ruby
|- resources
</pre>
<p>Download the Ruby runtimes from <a href="http://phusion.github.io/traveling-ruby/" target="_blank">Traveling Ruby</a>'s <a href="https://traveling-ruby.s3-us-west-2.amazonaws.com/list.html" target="_blank">S3 bucket</a> into the <code>resources</code> directory. I only needed the <a href="http://d6r77u77i8pq3.cloudfront.net/releases/traveling-ruby-20150715-2.2.2-osx.tar.gz" target="_blank">OSX version</a> for local development and the <a href="http://d6r77u77i8pq3.cloudfront.net/releases/traveling-ruby-20150715-2.2.2-linux-x86_64.tar.gz" target="_blank">Linux x86_64</a> version for AWS. My directory had these two files in it:</p>
<pre class="brush: shell">
- aws-lambda-ruby
|- hello_ruby
|- resources
|- traveling-ruby-20150715-2.2.2-linux-x86_64.tar.gz
|- traveling-ruby-20150715-2.2.2-osx.tar.gz
</pre>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/e4c12cb357f276f51ef771686a9fb4cad6df2162" target="_blank">Commit point</a></p>
<p>Create two new directories for assembling the project under OSX and Linux X86_64 like these:</p>
<pre class="brush: shell">
- aws-lambda-ruby
|- hello-2.0.0-linux-x86_64
|- hello-1.0.0-osx
|- hello_ruby
|- resources
</pre>
<p>Add a <a href="http://www.adomokos.com/2016/03/why-make.html" target="_blank">Makefile</a> to the project under the root directory, we want to automate all the different steps as early as possible. Create a Make target to package and run the code on OSX like this:</p>
<pre class="brush: shell">
run: ## Runs the code locally
@echo 'Run the app locally'
@echo '-------------------'
@rm -fr $(OSXDIR)
@mkdir -p $(OSXDIR)/lib/ruby
@tar -xzf resources/traveling-ruby-20150715-2.2.2-osx.tar.gz -C $(OSXDIR)/lib/ruby
@mkdir $(OSXDIR)/lib/app
@cp hello_ruby/lib/hello.rb $(OSXDIR)/lib/app/hello.rb
@cp resources/wrapper.sh $(OSXDIR)/hello
@chmod +x $(OSXDIR)/hello
@cd $(OSXDIR) && ./hello
</pre>
<p>Traveling Ruby suggests running the app through an executable shell script, that's what the <code>resources/wrapper.sh</code> file is:</p>
<pre class="brush: shell">
#!/bin/bash
set -e
# Figure out where this script is located.
SELFDIR="`dirname \"$0\"`"
SELFDIR="`cd \"$SELFDIR\" && pwd`"
# Run the actual app using the bundled Ruby interpreter.
exec "$SELFDIR/lib/ruby/bin/ruby" "$SELFDIR/lib/app/hello.rb"
</pre>
<p>If you have all the right files in the correct directories and your Makefile has the <code>run</code> target with the code above when you execute <code>make run</code>, this is what you should see in your terminal:</p>
<pre class="brush: shell">
$: make run
Run the app locally
-------------------
Hello from Ruby!
</pre>
<p>We ran the Ruby code with the Traveling Ruby packaged Ruby runtime, not with the locally installed Ruby, that was set up with a Ruby version manager.</p>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/1ff6a6bf0ea51f0e856f352f3509490d98841f28" target="_blank">Commit point</a></p>
<h4>3. Package the Code for AWS Lambda</h4>
<p>We need to package the code for AWS Lambda after running the app locally on OSX. You can easily check the Lambda runtime by running an AWS Lambda function with Python. Create a new AWS Lambda with the "hello-world-python" template with this Python code in it:</p>
<pre class="brush: ruby">
from __future__ import print_function
import json
import commands
print('Loading function')
def lambda_handler(event, context):
print(commands.getstatusoutput('cat /etc/issue'))
print(commands.getstatusoutput('uname -a'))
print(commands.getstatusoutput('pwd'))
</pre>
<p>There are plenty of tutorials out there to guide you through creating an AWS Lambda, please Google the solution if you don’t know what to do. When you run it, this is the information you should get:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/python_system_info_log_output.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/python_system_info_log_output.jpg" width="550" alt="python-system-info"></a></p>
<p>We will use Node.js to execute the code, place this JavaScript file in your <code>resources</code> directory with the name <code>index.js</code>:</p>
<pre class="brush: javascript">
process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT']
var exec = require('child_process').exec;
exports.handler = function(event, context) {
var command = `./hello`;
child = exec(command, {env: {'LD_LIBRARY_PATH': __dirname + '/lib'}}, function(error) {
// Resolve with result of process
context.done(error, 'Process complete!');
});
// Log process stdout and stderr
child.stdout.on('data', console.log);
child.stderr.on('data', console.error);
};
</pre>
<p>The index.handler will be invoked by Lambda, which will spawn a new child process by executing the <code>hello</code> shell script, which will run the Ruby code with Traveling Ruby.</p>
<p>The <code>package</code> Make target will assemble the directory for AWS Lambda and compress it into a zip file. This is how that code looks:</p>
<pre class="brush: shell">
LAMBDADIR=hello-1.0.0-linux-x86_64
...
package: ## Package the code for AWS Lambda
@echo 'Package the app for deploy'
@echo '--------------------------'
@rm -fr $(LAMBDADIR)
@rm -fr deploy
@mkdir -p $(LAMBDADIR)/lib/ruby
@tar -xzf resources/traveling-ruby-20150715-2.2.2-linux-x86_64.tar.gz -C $(LAMBDADIR)/lib/ruby
@mkdir $(LAMBDADIR)/lib/app
@cp hello_ruby/lib/hello.rb $(LAMBDADIR)/lib/app/hello.rb
@cp resources/wrapper.sh $(LAMBDADIR)/hello
@chmod +x $(LAMBDADIR)/hello
@cp resources/index.js $(LAMBDADIR)/
@cd $(LAMBDADIR) && zip -r hello_ruby.zip hello index.js lib/
@mkdir deploy
@cd $(LAMBDADIR) && mv hello_ruby.zip ../deploy/
@echo '... Done.'
...
</pre>
<p>I only list the content that I added, the <code>run</code> target is still in the Makefile but I omitted it here for brevity. When you execute <code>make package</code>, you should see the following output:</p>
<pre class="brush: shell">
$: make package
Package the app for deploy
--------------------------
... Done.
</pre>
<p>and a <code>hello_ruby.zip</code> file should be created in your <code>deploy</code> directory.</p>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/6721e9e8f7d15649385bdb9adf6593214ad6e250" target="_blank">Commit point</a></p>
<h4>4. Deploy the Packaged Ruby Code to AWS Lambda</h4>
<p>We created a <code>hello_ruby.zip</code> file in the previous section, let's deploy this zip file to AWS Lambda. Open the <a href="https://aws.amazon.com/console/" target="_blank">AWS Management Console</a> and select "Lambda" from the options. Your created Lambdas (if you had any) are listed here. Let’s start creating a new one by clicking on the "Create a Lambda function" button. Select the "node-exec" template:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/node_exec_template.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/node_exec_template.jpg" width="550" alt="node-exec"></a></p>
<p>Fill out the form as you see it in this screenshot:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/create_function.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/create_function.jpg" width="550" alt="create_function"></a></p>
<ol>
<li>Name it "HelloFromRuby"</li>
<li>Chose the option of "Upload a .ZIP file"</li>
<li>Use the <code>lambda_basic_execution</code> role, if you don’t have it, create it</li>
</ol>
<p>Confirm it and create the Lambda function.</p>
<p>Test the function by clicking on the blue "Test" button. You can accept the <code>HelloWorld</code> test template, those arguments are going to be ignored for now. You should see the following output:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/log_output.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/log_output.jpg" width="550" alt="log_output"></a></p>
<p>The string "Hello from Ruby!" is coming from the Ruby code executed by Traveling Ruby, just like we did locally.</p>
<p>Woohoo! Congrats, you’ve just created an AWS Lambda function with MRI Ruby.</p>
<h4>5. Use the AWS Command Line Interface to Publish an AWS Lambda Function</h4>
<p>Although creating a Lambda through the GUI works, it's not something I'd do in the long run. The steps of dropping and creating Lambdas can be automated through the AWS Command Line Interface, those scripts can be easily executed from a Make target. Let's add a new target to drop the already existing Lambda function:</p>
<p>(This blog post assumes you already know how to use the AWS Command Line Interface, you have it configured properly. There is good documentation around this, please look it up and set it up for yourself.)</p>
<pre class="brush: shell">
...
delete: ## Removes the Lambda
aws lambda delete-function --function-name HelloFromRuby
...
</pre>
<p>Your 'HelloFromRuby' Lambda function will be deleted when you run <code>make delete</code> in your terminal. Go back to the AWS Management Console to verify that your Lambda function got deleted.</p>
<p>Add your lambda with the following script in your Make file:</p>
<pre class="brush: shell">
...
create: ## Creates an AWS lambda function
aws lambda create-function \
--function-name HelloFromRuby \
--handler index.handler \
--runtime nodejs4.3 \
--memory 512 \
--timeout 10 \
--description "Saying hello from MRI Ruby" \
--role arn:aws:iam::___xyz___:role/lambda_basic_execution \
--zip-file fileb://./deploy/hello_ruby.zip
...
</pre>
<p>I masked the <code>role</code> argument, you need to find your correct "Role ARN" value under Security -> IAM -> Roles. You should look for it here:</p>
<p><a href="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/role_arn.jpg" target="_blank"><img src="https://raw.githubusercontent.com/adomokos/climb_that_mountain/master/resources/2016/06/role_arn.jpg" width="550" alt="role-arn"></a></p>
<p>If everything is configured properly, you should be able to create your AWS Lambda function by running <code>make create</code> in the terminal.</p>
<p>We can invoke the lambda from the command line as well, this Make target will do just that:</p>
<pre class="brush: shell">
...
invoke: ## Invoke the AWS Lambda in the command line
rm -fr tmp && mkdir tmp
aws lambda invoke \
--invocation-type RequestResponse \
--function-name HelloFromRuby \
--log-type Tail \
--region us-east-1 \
--payload '{"name":"John Adam Smith"}' \
tmp/outfile.txt \
| jq -r '.LogResult' | base64 -D
...
</pre>
<p>Please note, that I am using a lightweight JSON parser, <a href="https://stedolan.github.io/jq/" target="_blank">jq</a> to extract information from the response.
You should see the following response from AWS Lambda:</p>
<pre class="brush: shell; highlight: [4]">
START RequestId: e8c24c91-2165-11e6-a0b6-35430628271f Version: $LATEST
2016-05-24T04:13:46.403Z e8c24c91-2165-11e6-a0b6-35430628271f
Hello from Ruby!
END RequestId: e8c24c91-2165-11e6-a0b6-35430628271f
REPORT RequestId: e8c24c91-2165-11e6-a0b6-35430628271f
Duration: 214.12 ms
Billed Duration: 300 ms
Memory Size: 512 MB
Max Memory Used: 20 MB
</pre>
<p><a href="https://github.com/adomokos/aws-lambda-ruby/commit/7bf6c4e5d6f745d636dbdc6737db7f23a4371085" target="_blank">Commit point</a></p>
<p>This blog post guided you through the steps of running MRI Ruby on AWS lambda. In the <a href="http://www.adomokos.com/2016/06/using-ruby-with-activerecord-in-aws.html" target="_blank">upcoming post</a>, I'll show you how you can add gems and talk with an RDS instance from your Ruby code on AWS Lambda.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-13543165294696473952016-05-15T17:03:00.000-05:002016-05-15T17:03:32.652-05:00Currying in Haskell, Clojure, Ruby and JavaScript<p>I worked with a developer about a year ago, who had more experience with functional programming than I had. We worked on a Clojure project, and his deep Haskell background made him an expert on our team. This was especially revealing when we discussed partial function applications and currying. I was vaguely familiar with the concept, but I've never used them in any of the apps I've worked on.</p>
<p>Fast forward a year, after learning and playing with Haskell for a few months, I understand why: in Haskell, everything is curried. I repeat: everything. Even the function invocation is curried. In fact, you have to work hard if you want it differently. No wonder, currying was so obvious for that developer.</p>
<p>
<div style="text-align: center;">
<img src="https://github.com/adomokos/climb_that_mountain/raw/master/resources/2016/05/haskell.png" alt="haskell-logo" title="" style="width: 150px;" />
</div>
</p>
<p>Let's look at a simple example in Haskell:</p>
<pre class="brush: haskell">
-- You can type this in your own GHCi or try it in https://ghc.io/
let multiply x y = x * y
let double = multiply 2
let triple = multiply 3
double 3
triple 4
</pre>
<p>I created a multiply function that takes two arguments and multiplies them. In Haskell everything is curried, it's perfectly fine to invoke this function with only a single argument. What will I get back? <strong>Another function</strong>. This was the breakthrough for me: partially applying a function yields another function. Then I defined two other functions, one that passes in 2 to double, the other 3 to triple whatever argument I pass to it.</p>
<p>What I was amazed by this was the easiness and the natural nature of Haskell's currying through partial application.</p>
<p>Let's see how this simple example would look in Clojure.</p>
<pre class="brush: clojure">
(defn multiply [x y] (* x y))
(def double (partial multiply 2))
(def triple (partial multiply 3))
(double 3) ;; will yield 6
(triple 4) ;; will produce 12
</pre>
<p>This works, but yuck, I had to use a special language construct <code>partial</code> to signal, that I'll be partially applying the <code>multiply</code> function. Based on the Haskell example, my intuition was to use <code>defn</code> for the double and triple functions, but that tripped me over, it did not work. I had to "StackOverflow" it to realize, that the <code>def</code> binding is needed instead of <code>defn</code> to produce the partially applied function. This is a far cry from Haskell, where everything felt natural.</p>
<p>Although Ruby is a dynamically typed object-oriented language, it has many functional constructs that I enjoy using. I was curious if Ruby supports currying. To my surprise, it does. Look at the same example with partial functions and currying in Ruby.</p>
<pre class="brush: ruby">
multiply = -> (x, y) { x * y }
double = multiply.curry.(2)
triple = multiply.curry.(3)
double.(3) # will yield 6
triple.(4) # will produce 12
</pre>
Well, this works, but it's far from Haskell's obvious nature, where I did not have to use any kind of special keywords to achieve the same result.</p>
<p>Here is how I would write this with "programming by wishful thinking":</p>
<pre class="brush: ruby">
# This won't work
multiply = -> (x, y) { x * y }
double = multiply(2)
triple = multiply(3)
</pre>
<p>I am sure the Ruby language authors had a reason to use <code>curry</code> for partial applications, but it just did not feel natural. I have to learn and remember how to use it properly.</p>
<p>There are currying related npm packages in Node.js, but I have not found anything that's built into the language. Here is how the poor man's currying is done in JavaScript:</p>
<pre class="brush: javascript">
var multiply = function(x) {
return function(y) {
return x * y;
}
}
var double = multiply(2);
var triple = multiply(3);
double(3); // will yield 6
triple(4); // will produce 12
</pre>
I like JavaScript's "functions are first class citizens" nature, I am sure once ES6 or 7 gets widely adopted, it will be a language I'll enjoy using in the future.</p>
<p>Learning about currying in one language and using those concepts in another is an obvious benefit of <a href="https://pragprog.com/book/tpp/the-pragmatic-programmer" target="_blank">learning a programming language in every year</a>.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-63808273098988377582016-03-27T22:02:00.000-05:002016-04-07T07:21:22.047-05:00Why Make?<p>A new member joined our team a couple of weeks ago, and as we took him out for a cup of coffee on his first week, he asked me a question: "I’ve never seen this before and I wanted to ask you. Why are you using Make in your Ruby project?"</p>
<p>His question was legit coming from someone in the Ruby world, where we have <a href="https://github.com/ruby/rake" target="_blank">rake</a> for achieving the same goal. I had to think about the history there as I explained my reasoning behind using it.</p>
<p>As I started looking at other programming languages, <code>rake</code> wasn’t available for me. A couple of years ago I got pretty deep into node.js, and there were repetitive tasks I had to do, like dropping and rebuilding a database, running the tests, etc. I created a <code>script</code> directory and put separate scripts in it to accomplish all that.</p>
<p>For example, for the two tasks I mentioned above, I created two scripts:</p>
<pre class="brush: shell">
project_root
|- scripts
|- build_db.sh
|- run_tests.sh
</pre>
<p>This worked OK for a while, but when I contributed to the great testing framework <a href="https://github.com/mochajs/mocha" target="_blank">mocha.js</a>, I realized that the author of that module, <a href="http://tjholowaychuk.com/" target="_blank">T.J.</a>, just took these convenience scripts to another level by using <a href="https://github.com/mochajs/mocha/blob/master/Makefile" target="_blank">Make</a>.</p>
<p>I noticed he is using Make to run simple shell commands in a more elegant manner than I did with my scripts directory and shell scripts in it. I immediately started to adopt this practice.</p>
<p>As I started exploring other languages like Clojure, Erlang, Haskell, using Make was an obvious choice. It did not matter what language I used, dropping and building a database was the same task, regardless.</p>
<p>This practice came with me as I went back to Ruby as well. As I started working on larger, 3-4-year-old Rails apps, running rake tasks was a time-consuming exercise. Bundler had to load the whole world into memory before it could even evaluate what it had to do. This way, simple tasks that had nothing to do with Rails had an 8-10 seconds startup time. I did not think twice about firing up a Makefile to do the same in less than a second.</p>
<p>Of course, some of the religious Rails disciples dismissed this, but productivity over religion has a higher precedence for me.</p>
<p>A few weeks ago I found a <a href="http://marmelab.com/blog/2016/02/29/auto-documented-makefile.html" target="_blank">blog post</a> on how to add task description to Make targets. I updated my Makefiles, and now, when I run <code>make</code> in the terminal, this is what I see:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgN_7h0Uabv1-hZavzPTFuXmTtqzwgummumDdAdYsgI-1_NY6bvEGvwmmwLr4LAO_eMk_OQlsrJN-TRtGds7x8v5z7cZ6xwYFMFsTcArSoTUr3EwUhP8fv6FW-Hsrjn1K9XaBE-eBL3jKkj/s1600/make_output.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgN_7h0Uabv1-hZavzPTFuXmTtqzwgummumDdAdYsgI-1_NY6bvEGvwmmwLr4LAO_eMk_OQlsrJN-TRtGds7x8v5z7cZ6xwYFMFsTcArSoTUr3EwUhP8fv6FW-Hsrjn1K9XaBE-eBL3jKkj/s320/make_output.jpg" /></a></div>
<p>As I have not found a good make target generator, I created a <a href="https://gist.github.com/adomokos/2fd95840d59b19bbb3f4" target="_blank">gist</a> to get me rolling. Documentation and a sample target are a good way to get started. I even added a shell function to grab it for me:</p>
<pre class="brush: shell">
make-init() {
curl https://gist.githubusercontent.com/adomokos/2fd95840d59b19bbb3f4/raw/7b548cd3fda0dab958ecb0e0955fbadc1af6ef6e/Makefile > Makefile
}
</pre>
<p>Now, I only need to type <code>make-init</code> in the terminal and I have a Makefile to work with.</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-49950383873424805462016-02-19T13:10:00.000-06:002016-02-19T15:44:57.183-06:00Teaching Clojure to a 7-Year-Old<p>
I see my 7-year-old son as someone, who is deeply interested in anything computer related. I recall how he went through photos on my iPod Touch when he was only two years old. Then video games kicked in, and now he can make an 8-hour flight back to Europe without taking a break from a game on an iPad.
</p>
<p>
This is all nice and cool, but why don't we do something more useful with this passion?! He has been curious about programming, he practiced basic function calls via the game <a href="https://codecombat.com/" target="_blank">Code Combat</a>. I figured, let's take this to the next level and try programming.
</p>
<p>
I wanted to use a language that is easy to understand, but can be very powerful. I considered Python or JavaScript, but I figured his first real programming language should be a functional one, and the simple nature of LiSP made Clojure the obvious choice.
</p>
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLfdjcMNYRUnhyphenhyphenlH6aMWoMYAp_YeTpsiyAqw5buIcKaK4dyRXQ4WLjCnTUVgJt79j0mSlBOVuORriKsBpZP8ZQLjj3YNc9lv1V5qm_PmSKIkt7jQMHAcH65vHtdBwYI2W2F8Mtn5r0NyNU/s320/two_macs.jpg" /></div>
<p>
I wanted to find an editor that's easy to use. I was aware of <a href="http://lighttable.com/" target="_blank">LightTable</a>, but I've never tried it. We downloaded it on my wife's 11" MacBook Air and we jumped in.
</p>
<p>
What's really cool about LightTable is that you don't need to run a separate REPL, you could just write your Clojure expressions in a clj file, save it and by hitting <Cmd> + <Shift> + <Enter> the expressions are evaluated in line, right next to them. It's really the best tool for beginners.
</p>
<p>
We worked on a rectangular area calculator in our first session, since that's what he's been learning at school. This was our first expression:
<pre class="brush: clojure">
(defn area [x y] (* x y))
(area 2 3) # 6
</pre>
We tried different numbers, he was pumped when the correct number was printed in LightTable after the evaluation.
</p>
<p>
Our area function is for rectangles, but what if we wanted to calculate the square's area? We only had to pass one number to our calculator.<br>
Of course we could have done this:
<pre class="brush: clojure">
(area 4 4) # 16
</pre>
But this was not very elegant. I proposed creating a new function for <code>square-area</code> and calling <code>area</code> from that this way:
<pre class="brush: clojure">
(defn square-area [x]
(area x x))
(square-area [4]) # 16
</pre>
My goal here was to show him how one function can leverage the functionality of another function. He liked this a lot.
</p>
<p>
In our next session - as he constantly nudged me to continue his journey in Clojure programming land - I wanted to teach him something we could build upon: we learned about vectors, which is similar to Arrays in other programming languages.
</p>
<p>
We started out with listing his best friends:
<pre class="brush: clojure">
(def friends [:andy :james :tommy :ethan :elliot])
</pre>
I explained to him that these names are in order and they will always remain in order the way he defined it first. There are ways to explore the collection, for example pulling the first item from it. This is what we tried:
<pre class="brush: clojure">
(first friends) # :andy
</pre>
I asked him if he could get the last item of the vector. He thought about a bit and this is what he came up with:
<pre class="brush: clojure">
(last friends) # :elliot
</pre>
Excellent! Then I asked him how we could get the third item from the collection and he intuitively tried the function <code>third</code>, which does not exist. I showed him the <code>nth</code> function to do that. This is what he tried to get the third item:
<pre class="brush: clojure">
(nth friends 3) # :ethan
</pre>
But oh, it returned the fourth and not the third item. So we talked about the 0-based index, which he grasped, but did not make much sense to him.
</p>
<p>
I told him: "Imagine how great it would be to sort these names in alphabetical order. Do you know what verb would describe that operation?" He said "sort", so we gave it a try:
<pre class="brush: clojure">
(sort friends) # (:andy :elliot :ethan :james :tommy)
</pre>
We both smiled at how easy it was. Then I asked him if we could put the names in descending order. We were looking for the right word, and he came up with <code>backwards</code>. Well, it's close, so I asked him to look up synonyms for that word in Google. We both settled on <code>reverse</code>.
He tried this:
<pre class="brush: clojure">
(reverse friends) # (:elliot :ethan :tommy :james :andy)
</pre>
Oh-oh. This only put the original list into reverse order without sorting. It was obvious that we needed to sort it first and then reverse it. I helped him to write this:
<pre class="brush: clojure">
(reverse (sort friends)) # (:tommny :james :ethan :elliot :andy)
</pre>
</p>
<p>
We wrapped up our session by creating a vector with numbers.
<pre class="brush: clojure">
(def numbers [9 12 5 7 1])
</pre>
Based on what he learned earlier, he put them numbers in descending order:
<pre class="brush: clojure">
(reverse (sort numbers)) # (12 9 7 5 1)
</pre>
I showed him how we could use the <code>filter</code> and the <code>odd?</code> functions to filter out the odd numbers.
<pre class="brush: clojure">
(filter odd? numbers) # (9 5 7 1)
</pre>
He was able to sort the odd numbers by using the <code>sort</code> function:
<pre class="brush: clojure">
(sort (filter odd? numbers)) # (1 5 7 9)
</pre>
Picking the largest odd number was a tricky one, we had to look at the docs, as this did not work:
<pre class="brush: clojure">
(max (filter odd? numbers)) # (1 5 7 9)
</pre>
But this did:
<pre class="brush: clojure">
(apply max (filter odd? numbers)) # 9
</pre>
There was an easier way to get the largest odd number: reverse sorting the odd numbers and getting the first item. This is what we wrote:
<pre class="brush: clojure">
(first (reverse (sort (filter odd? numbers)))) # 9
</pre>
We both smiled when we saw the result.
</p>
<p>
This is the full code we wrote together:
<script src="https://gist.github.com/adomokos/a8fd769f0eacf46d8f47.js"></script>
</p>
<p>
We are going to spend half of our time going through the same function composition the next time. Then we might look at other collection types like maps and sets. I don't want to push him, but if he asks for it, we'll go as far as we can.
</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-76999332702616591362016-01-26T11:02:00.001-06:002016-02-01T16:41:56.738-06:00Vitamin, Medicine, Drug<p>
I worked with a good friend of mine on a product idea a few months ago. The number of disengaged employees at large enterprises is staggering, we both have witnessed this during our professional career. We tried measuring employee engagement and happiness by frequent, short surveys, providing a real-time engagement thermometer to management.
</p>
<p>
I talked with two investors about our idea and one of them told me this: "There are two types of products, vitamins and medicines. While vitamins are good to have for a company, it's not absolutely essential. The company can survive and even thrive without it. However, a medicine is a must have, companies can't live without it. I tend to invest in medicine-like product ideas, and I am sorry, yours is a vitamin. It can get big, but selling the idea will be hard."
</p>
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5V1kEJLl1GKBMfQ-g2l8BmaG9YB-aZX8dyFIsxM4Ld_NDy-h6UpEoZFHWScTYt-DMW78ASqF_1aQxzRinH5SNlPfw96R27w4bCxAgFnXlLDn26U3HdmKEI2qwCa7Ne5g08fvIP9mCKKYp/s320/vitamin.jpg" /></div>
<h4>The Vitamin</h4>
<p>
A vitamin product might be important for management, but the perceived value for the employee is unclear. Unless we were able to provide some kind of value for the person who fills out our survey, the employee would never be engaged.
</p>
<p>
I once had to track my hours on projects at a large enterprise just to provide data for the army of project managers to calculate actual project cost. Although this had value to the employer, it had very little value to me or my peers. We were constantly nudged by management to log the hours by the end of each week.<br>
Now, if the company is in consulting and the employee won't get paid unless she provides the hourly breakdown of billable hours, it's a different story. The employee has vested interest in providing the data, otherwise, she will never get paid.
</p>
<p style="font-style: italic;">Selling the Vitamin can take an army of sales people for cold calling prospects. The referral rate is low, users are not very engaged.</p>
<h4>The Medicine</h4>
<p>
The medicine product has real benefits for both the employee and the employer.<br>
I have witnessed SalesForce shifting from vitamin to medicine category before. Initially, it wasn't taken seriously by the sales teams, but as soon as it was leveraged for financial reporting, it became essential for the company.<br>
<a href="https://basecamp.com/" target="_blank">Basecamp</a> is another good example for medicine, which is adopted by the enterprise (mostly) through employee demand. I've read about people using Basecamp for their freelance projects, and when they join larger companies, they suggest this tool.<br>
<a href="https://github.com/" target="_blank">Github</a> is so good, it's pushing the boundaries of medicine. I have worked with many software engineers, however, I have never met a single sales person from Github trying to talk us into submitting our credit card and signing up for private repositories.
</p>
<p style="font-style: italic;">Medicine is easy to sell, users are recommending it to other potential customers. Companies with a medicine-like software have a smaller sales team. The reputation of the product is selling itself.</p>
<h4>The Drug</h4>
<p>
There is a third category this investor did not mention to me, but it exists out there. People are so hooked, they get angry when they don't have access to it. It's Facebook. The company <a href="http://www.dailymail.co.uk/sciencetech/article-3386068/Facebook-SABOTAGED-Android-app-annoyed-users-simply-switched-web-browser.html" target="_blank">did an exercise</a> just recently to train for the battle against Google, they made Facebook inaccessible for Android users to investigate what they would do. Users tried to restart the app on their mobile device a couple of times. When that did not work, they opened their browsers and logged on through that. No matter what, they did not want to miss anything that was happening with their friends.
</p>
<p>
Finding the drug is super hard. But medicine-like products can be invented, and vitamins can transition into medicine.
</p><p>
When you're searching for a new gig, or you are thinking about your new idea, skip the vitamins, and start out with the medicine.
</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-87046113625192676882015-11-01T20:13:00.001-06:002015-11-01T20:13:56.603-06:00Clojure API with Yesql, Migrations and More (Part 3.)<p>
We created a database with scripts, <a href="http://www.adomokos.com/2015/10/clojure-api-with-yesql-migrations-and.html" target="_blank">added migrations</a> and <a href="http://www.adomokos.com/2015/10/clojure-api-with-yesql-migrations-part2.html" target="_blank">communicated with the database</a> with the help of yesql in the previous posts. Please look at those first to get up to speed with this part.
</p>
<p>
In the final part of the series, we will serialize the data we pull from the database to JSON, and we will expose that data through an HTTP endpoint. We will also add logging to monitor the JDBC communication with the database.
<p>
It was about two years ago when I attended a conference and I sat down with a couple of friends one night for a chat. It was late in the evening, after a couple of beers they asked me what I was up to. I told them I am learning Clojure. They wanted to see it in action, we solved <a href="http://www.adomokos.com/2013/12/test-driving-fizzbuzz-kata-in-clojure.html" target="_blank">FizzBuzz</a> together. They liked it, but one question was lingering there: "can you build a web app with Clojure?". Of course!<br>
We started out as a console application, but the requirements have changed, we need to expose the data via an HTTP interface through JSON. I like to look at web frameworks as a "<a href="https://www.youtube.com/watch?v=WpkDN78P884" target="_blank">delivery mechanism</a>", progressing the idea this way follows that.
</p>
<p>
Use this <a href="https://github.com/adomokos/kashmir/commit/4da1a44dd767e1afa44e38984b7e742dfb14c1b0" target="_blank">commit</a> as a starting for this blog post. Rebuild the database by running <code>make build-db</code>.
</p>
<h4>Serializing the Data as JSON</h4>
<p>
We will use the <a href="https://github.com/dakrone/cheshire" target="_blank">cheshire</a> library to serialize the data to JSON. Let's modify the "project.clj" file this way, see my changes highlighted:
<pre class="brush: clojure; highlight: 5;">
...
:dependencies [[org.clojure/clojure "1.7.0"]
[org.postgresql/postgresql "9.4-1201-jdbc41"]
[yesql "0.5.1"]
[cheshire "5.5.0"]]
...
</pre>
The serialization should be taken care of by some kind of logic component. Let's write the test for this, place this content into your "test/kashmir/logic_test.clj" file:
<pre class="brush: clojure">
(ns kashmir.logic-test
(:require [clojure.test :refer :all]
[kashmir.logic :refer :all]
[cheshire.core :as json]))
(deftest find-member-by-id-test
(testing "returns a JSON serialized member record"
(let [member (first (json/parse-string (find-member 2) true))]
(is (= "Paul" (:first_name member))))))
</pre>
Let's add the function skeleton to see test errors and not Java failures. Put this in the "src/kashmir/logic.clj" file:
<pre class="brush: clojure">
(ns kashmir.logic)
(defn find-member [id] nil)
</pre>
Rebuild the database with the <code>make build-db</code> command. Running <code>lein test</code> should provide an output similar to this:
<pre class="brush: bash">
% lein test
lein test kashmir.data-test
lein test kashmir.logic-test
lein test :only kashmir.logic-test/find-member-by-id-test
FAIL in (find-member-by-id-test) (logic_test.clj:9)
returns a JSON serialized member record
expected: (= "Paul" (:first_name member))
actual: (not (= "Paul" nil))
Ran 4 tests containing 4 assertions.
1 failures, 0 errors.
Tests failed.
</pre>
Cheshire uses two main functions, <code>generate-string</code> to serialize and <code>parse-string</code> to deserialize data. We will have to serialize the data, please modify the "src/kashmir/logic.clj" file this way:
<pre class="brush: clojure; highlight: [2,3,6]">
(ns kashmir.logic
(:require [kashmir.data :as data]
[cheshire.core :as json]))
(defn find-member [id]
(json/generate-string (data/find-member id)))
</pre>
Run your tests again, all 4 should pass now.<br>
As you think about, the logic namespace is responsible for making sure the data component returned data, handling exceptions and validating user input. This is the part of the app I'd test the most.<br>
(<a href="https://github.com/adomokos/kashmir/commit/d654cdc2f57bafd699fd65a9344805bd7ed506ec" target="_blank">Commit point</a>.)
</p>
<h4>Exposing the Data with Compojure</h4>
<p>
<a href="https://github.com/weavejester/compojure" target="_blank">Compojure</a> is our go-to tool when it comes to building a web interface without much ceremony. Let's add it to our "project.clj" file:
<pre class="brush: clojure; highlight: [9, 10, 14, 15, 16, 20, 21]">
(defproject kashmir "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.7.0"]
[org.postgresql/postgresql "9.4-1201-jdbc41"]
[yesql "0.5.1"]
[compojure "1.4.0"]
[ring/ring-defaults "0.1.5"]
[cheshire "5.5.0"]]
:clj-sql-up {:database "jdbc:postgresql://kashmir_user:password@localhost:5432/kashmir"
:deps [[org.postgresql/postgresql "9.4-1201-jdbc41"]]}
:ring {:handler kashmir.handler/app}
:plugins [[clj-sql-up "0.3.7"]
[lein-ring "0.9.7"]]
:main ^:skip-aot kashmir.core
:target-path "target/%s"
:profiles {:uberjar {:aot :all}
:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
[ring-mock "0.1.5"]]}})
</pre>
We also need to add a "src/kashmir/handle.clj" file, that will handle the different web requests:
<pre class="brush: clojure;">
(ns kashmir.handler
(:require [compojure.core :refer :all]
[compojure.route :as route]
[ring.middleware.defaults :refer [wrap-defaults api-defaults]]
[kashmir.logic :as logic]))
(defroutes api-routes
(GET "/" [] "Hello World")
(GET "/members/:id{[0-9]+}" [id]
{:status 200
:headers {"Content-Type" "application/json; charset=utf-8"}
:body (logic/find-member (read-string id))})
(route/not-found "Not Found"))
(def app
(wrap-defaults api-routes api-defaults))
</pre>
Fire up the server with the <code>lein ring server-headless</code> command. Open up a new terminal window, and request the member with ID 2 using the curl command: <code>curl -i http://localhost:3000/members/2</code>. You should see something like this:
<pre class="brush: bash;">
% curl -i http://localhost:3000/members/2
HTTP/1.1 200 OK
Date: Thu, 15 Oct 2015 17:31:44 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 123
Server: Jetty(7.6.13.v20130916)
[{"id":2,"first_name":"Paul","last_name":"McCartney",
"email":"pmccartney@beatles.com","created_at":"2015-10-15T16:50:03Z"}]%
</pre>
The <code>-i</code> switch for curl will print out both the header and the body of the response.<br>
(<a href="https://github.com/adomokos/kashmir/commit/8cc9b9ae500e0493a80866f24e06193fd25566f3" target="_blank">Commit point</a>.)
</p>
<h4>Using Ring Response</h4>
<p>
The way we are generating the response is too verbose, we are explicitly setting the status, the headers and the body. There are ring helpers we can take advantage of, making this a lot shorter.<br>
Change the "src/kashmir/handler.clj" file content to this (highlighted rows will designate changes):
<pre class="brush: clojure; highlight: [5, 11]">
(ns kashmir.handler
(:require [compojure.core :refer :all]
[compojure.route :as route]
[ring.middleware.defaults :refer [wrap-defaults api-defaults]]
[ring.util.response :as rr]
[kashmir.logic :as logic]))
(defroutes api-routes
(GET "/" [] "Hello World")
(GET "/members/:id{[0-9]+}" [id]
(rr/response (logic/find-member (read-string id))))
(route/not-found "Not Found"))
(def app
(wrap-defaults api-routes api-defaults))
</pre>
Fire up the server, run the curl request, everything should still work the same.<br>
(<a href="https://github.com/adomokos/kashmir/commit/c9e9bb433d336a31ef3a9c00eaa89dc9b297fda9" target="_blank">Commit point</a>.)
</p>
<h4>Stubbing out Data Access in Logic Tests</h4>
<p>
Hitting the database for the logic function is feasible, but it won't buy you all that much. You can stub out your database call with Clojure's <code>with-redefs</code> function. You need to define a function that returns the value the data access function would return.
</p>
</p>
Modify the "test/kashmir/logic_test.clj" file this way:
<pre class="brush: clojure; highlight: [4, 9]">
(ns kashmir.logic-test
(:require [clojure.test :refer :all]
[kashmir.logic :refer :all]
[kashmir.data :as data]
[cheshire.core :as json]))
(deftest find-member-by-id-test
(testing "returns a JSON serialized member record"
(with-redefs [data/find-member (fn [id] [{:first_name "Paul"}])]
(let [member (first (json/parse-string (find-member 2) true))]
(is (= "Paul" (:first_name member)))))))
</pre>
Now, stop your Postgres database server and run this test, it should pass as it's not hitting the database, it purely tests the hash serialization.<br>
(<a href="https://github.com/adomokos/kashmir/commit/debc055da395f3d18edad13fd9d69b0cdeb0cfe8" target="_blank">Commit point</a>.)
</p>
<h4>Adding JDBC Logging</h4>
<p>
Our solution works well as it is, however, we don't see what kind of SQL statements are executed against the database. Turning on logging in Postgres is one option, but monitoring JDBC within our application is prefereable. We will use the <a href="https://github.com/arthurblake/log4jdbc" target="_blank">log4jdbc</a> library to log jdbc activities. This library is using the <a href="http://www.slf4j.org/download.html" target="_blank">Simple Logging Facade For Java</a> library, you need to add that jar file to the project.
</p>
<p>
Download the <a href="http://www.slf4j.org/dist/slf4j-1.7.12.tar.gz">slf4j jar file</a> and add it to the project's lib directory. Then modify the "project.clj" file this way:
<pre class="brush: clojure; highlight: [6, 10]">
[yesql "0.5.1"]
[compojure "1.4.0"]
[ring/ring-defaults "0.1.5"]
[cheshire "5.5.0"]]
[cheshire "5.5.0"]
[com.googlecode.log4jdbc/log4jdbc "1.2"]]
:clj-sql-up {:database "jdbc:postgresql://kashmir_user:password@localhost:5432/kashmir"
:deps [[org.postgresql/postgresql "9.4-1201-jdbc41"]]}
:ring {:handler kashmir.handler/app}
:resource-paths ["lib/slf4j-simple-1.7.12.jar"]
:plugins [[clj-sql-up "0.3.7"]
[lein-ring "0.9.7"]]
:main ^:skip-aot kashmir.core
</pre>
You need to configure slf4j, you can do that by adding this content to the "resources/log4j.properties" file:
<pre class="brush: bash;">
# the appender used for the JDBC API layer call logging above, sql only
log4j.appender.sql=org.apache.log4j.ConsoleAppender
log4j.appender.sql.Target=System.out
log4j.appender.sql.layout=org.apache.log4j.PatternLayout
log4j.appender.sql.layout.ConversionPattern= \u001b[0;31m (SQL)\u001b[m %d{yyyy-MM-dd HH:mm:ss.SSS} \u001b[0;32m %m \u001b[m %n
# ==============================================================================
# JDBC API layer call logging :
# INFO shows logging, DEBUG also shows where in code the jdbc calls were made,
# setting DEBUG to true might cause minor slow-down in some environments.
# If you experience too much slowness, use INFO instead.
log4jdbc.drivers=org.postgresql.Driver
# Log all JDBC calls except for ResultSet calls
log4j.logger.jdbc.audit=FATAL,sql
log4j.additivity.jdbc.audit=false
# Log only JDBC calls to ResultSet objects
log4j.logger.jdbc.resultset=FATAL,sql
log4j.additivity.jdbc.resultset=false
# Log only the SQL that is executed.
log4j.logger.jdbc.sqlonly=FATAL,sql
log4j.additivity.jdbc.sqlonly=false
# Log timing information about the SQL that is executed.
log4j.logger.jdbc.sqltiming=FATAL,sql
log4j.additivity.jdbc.sqltiming=false
# Log connection open/close events and connection number dump
log4j.logger.jdbc.connection=FATAL,sql
log4j.additivity.jdbc.connection=false
</pre>
Finally, you need to modify the "src/kashmir/data.clj" file to use the logger Postgres connection:
<pre class="brush: clojure; highlight: [4, 5]">
(:require [yesql.core :refer [defqueries]]
[clojure.java.jdbc :as jdbc]))
(def db-spec {:classname "net.sf.log4jdbc.DriverSpy"
:subprotocol "log4jdbc:postgresql"
:subname "//localhost:5432/kashmir"
:user "kashmir_user"
:password "password1"})
</pre>
Now when you run the tests or hit the HTTP endpoint with cURL, you should see the JDBC logs in the terminal:
<pre class="brush: bash;">
lein test kashmir.data-test
[main] INFO jdbc.connection - 1. Connection opened
[main] INFO jdbc.audit - 1. Connection.new Connection returned
[main] INFO jdbc.audit - 1. PreparedStatement.new PreparedStatement returned
[main] INFO jdbc.audit - 1. Connection.prepareStatement(SELECT *
FROM members
WHERE id = ?) returned net.sf.log4jdbc.PreparedStatementSpy@51dbed72
[main] INFO jdbc.audit - 1. PreparedStatement.setObject(1, 2) returned
[main] INFO jdbc.sqlonly - SELECT * FROM members WHERE id = 2
...
</pre>
(<a href="https://github.com/adomokos/kashmir/commit/81d40aa906908e815f16f53e00701124a6701405" target="_blank">Commit point</a>.)
</p>
<p>
As you can see, the log can be verbose. The easiest way I found to turn off logging is changing the <code>log4jdbc:postgresql</code> subprotocol back to the original value: <code>postgresql</code>.<br>
(<a href="https://github.com/adomokos/kashmir/commit/2300a4a90caf3599fb2c23337a3b60dcda931305" target="_blank">Commit point</a>.)
</p>
<p>
This last step concludes the series. We set up a database build process, added migrations and seed data to it. We separated SQL from Clojure by using the yesql library. We added testing with mocking to make sure our code is working properly. We exposed the data as JSON through an HTTP endpoint and we added JDBC logging to the project to monitor the communication with the database.
</p>
<p>
I hope you will find this exercise helpful. Good luck building your database backed Clojure solution!
</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-49257300123168526202015-10-31T10:38:00.000-05:002015-11-01T20:16:04.699-06:00Clojure API with Yesql, Migrations and More (Part 2.)<p>
In the <a href="http://www.adomokos.com/2015/10/clojure-api-with-yesql-migrations-and.html" target="_blank">previous article</a>, we started working on <a href="https://github.com/adomokos/kashmir" target="_blank">kashmir</a>, a Clojure project that interacts with a database, and exposes the data through a JSON HTTP endpoint.<br>
In this post we'll seed the database with some test data, add <a href="https://github.com/krisajenkins/yesql" target="_blank">yesql</a> as our DB communication tool, at the end we will cover testing.
</p>
<h4>Adding Seed Data</h4>
<p>
Use <a href="https://github.com/adomokos/kashmir/commit/cfe053fbc712e7125b80a440f8578d02a5ea6b0b" target="_blank">this commit</a> as your starting point for this exercise. Rebuild your database by running <code>make build-db</code> to make sure you have no records in the tables. Create a new file in resources/seeds.sql and add the following content to it:
<pre class="brush: sql">
INSERT INTO bands(name) VALUES ('The Beatles');
INSERT INTO bands(name) VALUES ('The Doors');
INSERT INTO members(first_name, last_name, email)
VALUES ('John', 'Lennon', 'jlennon@beatles.com');
INSERT INTO members(first_name, last_name, email)
VALUES ('Paul', 'McCartney', 'pmccartney@beatles.com');
INSERT INTO members(first_name, last_name, email)
VALUES ('George', 'Harrison', 'gharrison@beatles.com');
INSERT INTO members(first_name, last_name, email)
VALUES ('Ringo', 'Starr', 'rstarr@beatles.com');
INSERT INTO bands_members(band_id, member_id) VALUES(1, 1);
INSERT INTO bands_members(band_id, member_id) VALUES(1, 2);
INSERT INTO bands_members(band_id, member_id) VALUES(1, 3);
INSERT INTO bands_members(band_id, member_id) VALUES(1, 4);
</pre>
This will create 2 band and 4 member records. It will also associate the members of The Beatles with their band record. We will insert these records through Postgres' command line tool. Let's add this to our build-db target in our Makefile:
<pre class="brush: xml; highlight: 6">
...
build-db:
dropdb --if-exists --username $(USER) $(DBNAME) -h $(HOST) -p $(PORT)
createdb --username $(USER) $(DBNAME) -h $(HOST) -p $(PORT)
lein clj-sql-up migrate
psql -U $(USER) -d $(DBNAME) --file resources/seeds.sql > /dev/null
</pre>
We added <code>< /dev/null</code> to this line, we are not interested seeing how many records got inserted into the tables.
When you run <code>make build-db</code> you should have the seed data inserted into your database.<br>
(<a href="https://github.com/adomokos/kashmir/commit/2990ad682f3d934275941b300aa453d2f116f738" target="_blank">Commit point</a>.)
</p>
<h4>Talking to the Database with Yesql</h4>
<p>
The natural way to communicate with a database in Clojure is using <a href="https://github.com/clojure/java.jdbc" target="_blank">java.jdbc</a>. However, the spaghetti SQL is hard to understand, and mixing of Clojure code with SQL could make it a mess very quickly. I found the fantastic tool <a href="https://github.com/krisajenkins/yesql" target="_blank">yesql</a> a few weeks ago and it was just what I needed: an easy way to separate SQL from Clojure. Let's add yesql and the Postgres jdbc driver to the project by modifying the project.clj file this way:
<pre class="brush: clojure; highlight: [4,5]">
(defproject kashmir "0.1.0-SNAPSHOT"
...
:dependencies [[org.clojure/clojure "1.7.0"]
[org.postgresql/postgresql "9.4-1201-jdbc41"]
[yesql "0.5.1"]]
...)
</pre>
Create a new directory called "sql" under "src/kashmir". Create a new SQL file in this directory called "data.sql". Add these two queries to it:
<pre class="brush: sql;">
-- name: find-member-by-id
-- Find the member with the given ID(s).
SELECT *
FROM members
WHERE id = :id
-- name: count-members
-- Counts the number of members
SELECT count(*) AS count
FROM members
</pre>
The line in this SQL file that begins with <code>-- name:</code> has special significance. Yesql will create data access functions with the name you define there.<br>
Add a new Clojure file under "src/kashmir" called "data.clj", this file will hold the data access functions. Add the following code to it:
<pre class="brush: clojure">
(ns kashmir.data
(:require [yesql.core :refer [defqueries]]
[clojure.java.jdbc :as jdbc]))
(def db-spec {:classname "org.postgresql.Driver"
:subprotocol "postgresql"
:subname "//localhost:5432/kashmir"
:user "kashmir_user"
:password "password1"})
(defqueries "kashmir/sql/data.sql"
{:connection db-spec})
</pre>
I am a bit unhappy with duplicating the Postgres connection information here, I'll leave you to set up the DB connection in the project.clj file.<br>
Fire up the REPL to see if this works (you can find my input highlighted below):
<pre class="brush: bash; highlight: [13, 15, 17]">
% lein repl
nREPL server started on port 55527 on host 127.0.0.1 - nrepl://127.0.0.1:55527
REPL-y 0.3.7, nREPL 0.2.10
Clojure 1.7.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
kashmir.core=> (require '[kashmir.data :refer :all])
nil
kashmir.core=> (count-members)
({:count 4})
kashmir.core=> (find-member-by-id {:id 2})
({:id 2, :first_name "Paul", :last_name "McCartney", :email "pmccartney@beatles.com", :created_at #inst "2015-10-14T19:59:48.905474000-00:00"})
kashmir.core=>
</pre>
Fantastic! We can talk to the database through yesql based on the SQL scripts defined in "src/kashmir/sql/data.sql" file.<br>
(<a href="https://github.com/adomokos/kashmir/commit/fe6cba24ecaf163644a8ba7507cc74088ccd137a" target="_blank">Commit point</a>.)
</p>
<h4>Adding Tests</h4>
<p>
Although our application does not have much logic just yet, I'd like to show you how you could start writing automated tests. Create a new test file under "test/kashmir/data_test.clj". Add the following code to it:
<pre class="brush: clojure">
(ns kashmir.data-test
(:require [clojure.test :refer :all]
[kashmir.data :refer :all]))
(deftest count-members-test
(testing "there are 4 members"
(is (= 4 (-> (count-members) first :count)))))
</pre>
Remove the failing test in "test/kashmir/core_test.clj" file:
<pre class="brush: clojure">
(ns kashmir.core-test
(:require [clojure.test :refer :all]
[kashmir.core :refer :all]))
</pre>
Run the tests by invoking <code>lein test</code> and you should see the one and only test passing:
<pre class="brush: bash">
% lein test
lein test kashmir.data-test
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
</pre>
(<a href="https://github.com/adomokos/kashmir/commit/0d8ac36f564c6894f91413d7e7295c09b8753642" target="_blank">Commit point</a>.)
</p>
<h4>Finding a Member</h4>
<p>
Yesql needs a hash even when a record is looked up by an ID. This is how you invoke the yesql generated function: <code>(find-member-by-id {:id 2})</code>. We should keep the data access interface unaware of this implementation detail. Let's find a member by an ID this way: <code>(find-member 2)</code>. Write the test for this in test/kashmir/data_test.clj:
<pre class="brush: clojure">
...
(deftest find-member-by-id-test
(testing "finds PM with id 2"
(is (= "Paul" (-> (find-member 2) first :first_name)))))
</pre>
This is the code implementation of it in "src/kashmir/data.clj":
<pre class="brush: clojure;">
...
(defn find-member [id]
(find-member-by-id {:id id}))
</pre>
Both of the tests should pass now.<br>
(<a href="https://github.com/adomokos/kashmir/commit/ca9dfd94daabb44b3a6777b978531ba405b1fd66" target="_blank">Commit point</a>.)
</p>
<h4>Adding a Member</h4>
<p>
Reading data with yesql is simple, but adding records and testing that over and over can be more challenging. The database has to be reset to its original state after each test run. You have two options here:
<ul>
<li>truncate all the tables after each test,</li>
<li>roll back the INSERT transactions.</li>
</ul>
The way you can truncate all the tables is a blog post in itself. Unfortunately the Clojure community has not created a <a href="https://github.com/DatabaseCleaner/database_cleaner" target="_blank">DatabaseCleaner</a> project we love so much in the Ruby world just yet. Let's use the roll-back feature of the INSERT transaction for our tests in the examples.
</p>
<p>
When you create a member, you need to associate that member with a band. In fact, a member can not be added to the database without a band. A hash with all the member data and the band name will be the arguments to this create function.<br>
Let's write the test first in the "test/kashmir/data_test.clj" file:
<pre class="brush: clojure">
...
(deftest create-member-test
(testing "adds a member to the DB"
(let [member {:first_name "Jim" :last_name "Morrison" :email "jmorrison@doors.com"}]
(is (= 1 (create-member! member "The Doors"))))))
</pre>
Let's write the simplest code that could possibly work. First, we need to add the INSERT SQL statements to "src/kashmir/data.sql". This is what those look like:
<pre class="brush: sql">
...
-- name: find-band-by-name
-- Finds a band record based on the provided name
SELECT *
FROM bands
WHERE name = :name
-- name: create-member-raw!
-- Adds a new member with the bands_members association
WITH inserted AS (
INSERT INTO members (first_name, last_name, email)
VALUES (:first_name, :last_name, :email)
RETURNING id
)
INSERT INTO bands_members (member_id, band_id)
SELECT inserted.id, :band_id FROM inserted
</pre>
As I was writing this blog post, I researched how I could insert records into different tables with one SQL statement. Using stored procedure or function would be an easy choice, but that's too heavy for what we need. I found <a href="http://rob.conery.io/2015/02/09/inserting-using-new-record-postgres/" target="_blank">this blog post</a> by Rob Conery. He shows how CTEs (Common Table Expressions) can be used to insert and reuse the created record in a subsequent insert. That's what you see in the second SQL command. By using this solution, the Clojure code will be small, as the database write functionality is delegated to PostgreSQL.<br>
This is what the data logic will look like in the "src/kashmir/data.clj" file:
<pre class="brush: clojure">
...
(defn create-member!
([member band-name]
(let [band-id (-> (find-band-by-name {:name band-name})
first
:id)]
(create-member-raw! (conj member {:band_id band-id})))))
</pre>
The "-raw" postfix was used for the function that gets generated by yesql. We could have created an alias, but I liked this kind of naming-convention.<br>
When you run the test it won't error out, but one of the tests will fail. It has more than 4 total records in the members table. Absolutely, the database was not restored to its default state. Let's take care of that! We will insert the record, but we will roll back the transaction once the test is complete, leaving the database in it's original, default state.<br>
Add/modify the highlighted lines in your "src/kashmir/data.clj" file:
<pre class="brush: clojure; highlight: [4,5,6,7,11,12]">
...
(defn create-member!
([member band-name]
(jdbc/with-db-transaction [tx db-spec]
(create-member! member band-name tx)))
([member band-name tx]
(let [band-id (-> (find-band-by-name {:name band-name})
first
:id)]
(create-member-raw! (conj member {:band_id band-id})
{:connection tx}))))
</pre>
And finally, initialize and roll back the transaction from the test. Change the highlighted lines in "test/kashmir/data_test.clj" this way:
<pre class="brush: clojure; highlight: [4,8,9,12]">
(ns kashmir.data-test
(:require [clojure.test :refer :all]
[kashmir.data :refer :all]
[clojure.java.jdbc :as jdbc]))
...
(deftest create-member-test
(jdbc/with-db-transaction [tx db-spec]
(jdbc/db-set-rollback-only! tx)
(testing "adds a member to the DB"
(let [member {:first_name "Jim" :last_name "Morrison" :email "jmorrison@doors.com"}]
(is (= 1 (create-member! member "The Doors" tx)))))))
</pre>
Rebuild your database and run your tests. You should see the <code>0 failures, 0 errors.</code> message. Do it many times, the tests should always pass.<br>
(<a href="https://github.com/adomokos/kashmir/commit/5f4e280453dc96b0ca97e9774ef0d240c3b6872a" target="_blank">Commit point</a>.)
</p>
<h4>One Last Refactoring</h4>
<p>
I am unhappy with the <code>create-member!</code> function. The way we are looking up the band by its name is inelegant, I feel we could do better. Since we have one band record by name, when we call <code>find-band-by-name</code>, we should get back one single hash and not a lazy-seq with a hash in it. Let's refactor to that! First, we'll renamed the yesql generated function to <code>find-band-by-name-raw</code> in the "src/kashmir/sql/data.sql" file:
<pre class="brush: sql; highlight: 3;">
...
-- name: find-band-by-name-raw
-- Finds a band record based on the provided name
SELECT *
FROM bands
WHERE name = :name
</pre>
Let's refactor the actual code like this in "src/kashmir/data.clj":
<pre class="brush: clojure; highlight: [3,4,11,12]">
...
(defn find-band-by-name [name]
(first (find-band-by-name-raw {:name name})))
(defn create-member!
([member band-name]
(jdbc/with-db-transaction [tx db-spec]
(create-member! member band-name tx)))
([member band-name tx]
(let [band-id (:id (find-band-by-name band-name))]
(create-member-raw! (conj member {:band_id band-id})
{:connection tx}))))
</pre>
I rebuilt the db, ran the tests and everything passed.<br>
(<a href="https://github.com/adomokos/kashmir/commit/4da1a44dd767e1afa44e38984b7e742dfb14c1b0" target="_blank">Commit point</a>.)
</p>
<p>
You could say this is only the "happy path", what if the band name is incorrect and no band will be found. This will blow up somewhere. Absolutely! You need to do exception handling and error checking. I wanted to keep my examples simple, so others coming to Clojure can benefit from the simplified code.
</p>
<p>
This last refactoring concludes the second part of the series. In the <a href="http://www.adomokos.com/2015/11/clojure-api-with-yesql-migrations-part3.html" target="_blank">final session</a> we will add logging to jdbc to monitor how yesql communicates with the database. We will also expose the data as JSON through an HTTP endpoint.
</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-80454577885318370202015-10-31T10:23:00.000-05:002015-11-01T20:14:53.013-06:00Clojure API with Yesql, Migrations and More (Part 1.)<p>
I've found endless books and articles explaining core ideas and the building blocks of the Clojure programming language. They show you how to use the different data structures, they have good examples for little tasks, like reading and parsing a CSV file, but books or articles that walk you through an example of building a comprehensive solution is hard to find.<br>
I always liked writings that showed me how to build an app. I would learn many aspects of a language, get familiar with tools, and most of all, I would build something that could serve as foundation for my future projects.<br>
I am planning to do just that with this series of blog posts. I'd like to show you how to:
<ul>
<li>Set up your database environment with scripts</li>
<li>Manage database changes through migrations</li>
<li>Test the different components</li>
<li>Stub out function calls you don't need for testing</li>
<li>Add logging to monitor the database communication</li>
<li>And all this in Clojure!</li>
</ul>
By the end of these blog posts you will be able to expose database through Clojure libraries as JSON data with HTTP endpoints.
</p>
<p>
I have following presumptions:
<ul>
<li>You have Clojure installed (I have version 1.7.0 at the time of writing)</li>
<li>We'll use PostgreSQL (mine is 9.4.4)</li>
<li>I am using OSX (10.10.5)
</ul>
The name of the app is "kashmir", you can find the final solution in <a href="https://github.com/adomokos/kashmir" target="_blank">this public repo</a>. I will link specific commit points to the blog posts, this way you can join this tutorial at any point you want. Let's dive in!
</p>
<h4>The Data Model</h4>
<p>
The data model is simple, it has only 3 tables. The <code>members</code> table lists the various band members, the <code>bands</code> table lists all the bands those members belong to, and the <code>bands_members</code> table is used to map the members to their bands.<br>
This is what it looks like:
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXm5eJ9DuqWffOKjlgYwpd3eC0KxVnEsaD230kuBmu9hXv76KUu5BUIZ0X_Wl4fEmrYpxcC1QYApMK0gkQuQ9jOwMY-PYfOZ6rQfSzouqQiWp1ZWNA_CRIgQ5IJ5A3paihjnxlPSVTigyH/s1600/bands_members.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXm5eJ9DuqWffOKjlgYwpd3eC0KxVnEsaD230kuBmu9hXv76KUu5BUIZ0X_Wl4fEmrYpxcC1QYApMK0gkQuQ9jOwMY-PYfOZ6rQfSzouqQiWp1ZWNA_CRIgQ5IJ5A3paihjnxlPSVTigyH/s400/bands_members.png" /></a></div>
</p>
<h4>Creating the Database User</h4>
<p>
I use the excellent <a href="http://pgcli.com/" target="_blank">pgcli</a> tool as my command line interface for Postgres. It has code completion, table name suggestion features, it's your <code>psql</code> tool on steorid. If you don't have it, grab it through homebrew.
Create a DB user called "kashmir_user" and allow this user to create DBs. This is how you do it in the command line, all the inputs are highlighted:
<pre class="brush: bash; highlight: [6,10]">
% pgcli postgres
Version: 0.19.1
Chat: https://gitter.im/dbcli/pgcli
Mail: https://groups.google.com/forum/#!forum/pgcli
Home: http://pgcli.com
postgres> CREATE USER kashmir_user PASSWORD 'password1';
CREATE ROLE
Command Time: 0.000s
Format Time: 0.000s
postgres> ALTER USER kashmir_user CREATEDB;
ALTER ROLE
Command Time: 0.000s
Format Time: 0.000s
postgres>
</pre>
</p>
<h4>Initializing the Project</h4>
<p>
Generate the new project skeleton by runnig the <code>lein new app kashmir</code> command in the terminal. You should have a skeleton app project that looks like <a href="https://github.com/adomokos/kashmir/commit/70a6a45c6372bf5ab5cf02ec5c6e4a4853d44a77" target="_blank">this</a>. When you run <code>lein run</code>, you see "Hello, World!", and when you run the tests you see the 1 failure:
<pre class="brush: bash">
% lein test
lein test kashmir.core-test
lein test :only kashmir.core-test/a-test
FAIL in (a-test) (core_test.clj:7)
FIXME, I fail.
expected: (= 0 1)
actual: (not (= 0 1))
Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Tests failed.
</pre>
</p>
<h4>Creating the Database</h4>
<p>Database drop and create operations should be scripted. You can use <code>rake db:drop</code> and <code>rake db:create</code> in Rails, we should have something similar here. You can use the Postgres command line tools to create and drop databases with the <code>createdb</code> and <code>dropdb</code> commands. The <code>--if-exists</code> switch helps you when you're running it for the first time, the command won't error out if the database does not exist.<br>
The easiest way to create executable tasks is with a Makefile. Create a new file called <code>Makefile</code> in your project root and add this to it:
<pre class="brush: xml">
DBNAME=kashmir
USER=kashmir_user
PORT=5432
HOST=localhost
PGPASSWORD=password1
# Builds the DB by dropping and recreating it
build-db:
dropdb --if-exists --username $(USER) $(DBNAME) -h $(HOST) -p $(PORT)
createdb --username $(USER) $(DBNAME) -h $(HOST) -p $(PORT)
</pre>
We set up variables in the Makefile, it will be easy to change these values later, you are also adhering to good DRY principles.<br>
Run the Make target by typing the command <code>make build-db</code> in the terminal. You can run this as many times as you want, it will drop and recreate the empty database for you.<br>
(<a href="https://github.com/adomokos/kashmir/commit/8fb99ec0b0f5bd21322c5ee1c8f270a486a49a55" target="_blank">Commit point</a>.)
</p>
<h4>Running Migrations</h4>
<p>
The best way to implement changes in a database is through reversible migrations. I usually use the great <a href="https://github.com/ckuttruff/clj-sql-up" target="_blank">clj-sql-up</a> migration tool for that. Let's add it to the project.clj file:
<pre class="brush: clojure; highlight: [3,4,5]">
(defproject kashmir "0.1.0-SNAPSHOT"
...
:clj-sql-up {:database "jdbc:postgresql://kashmir_user:password@localhost:5432/kashmir"
:deps [[org.postgresql/postgresql "9.4-1201-jdbc4"]]}
:plugins [[clj-sql-up "0.3.7"]])
</pre>
Run the command <code>lein clj-sql-up create create-members</code> to generate your first migration. This should create a new file in the "migrations" directory. Open up that file and add your migration SQL to it:
<pre class="brush: clojure">
(defn up []
["CREATE TABLE members(id SERIAL PRIMARY KEY,
first_name varchar(50) NOT NULL,
last_name varchar(50) NOT NULL,
email varchar(50) NOT NULL,
created_at timestamp NOT NULL default CURRENT_TIMESTAMP)"
"CREATE INDEX idx_members_id ON members(id)"
"CREATE UNIQUE INDEX idx_email_unique ON members(email)"])
(defn down []
["DROP TABLE members"])
</pre>
Test your SQL by running <code>lein clj-sql-up migrate</code> in the terminal. I would recommend looking at the database to make sure the first table, "members" got created properly. Open up pgcli and run <code>\dt</code> from the pgcli prompt. You should see two tables listed there:
<ul>
<li>clj_sql_migrations</li>
<li>members</li>
</ul>
The table "clj_sql_migrations" is used to track the actual version of your database, it's the metadata for clj-sql-up to run the migrations.
Let's add the "bands" and "bands_members" tables as well, create a new migration file with the clj-sql-up generator: <code>lein clj-sql-up create create-bands</code>. Open up the migrations/*-create-bands.clj file and add this SQL:
<pre class="brush: clojure">
(defn up []
["CREATE TABLE bands(id SERIAL PRIMARY KEY,
name varchar(50) NOT NULL,
created_at timestamp NOT NULL default CURRENT_TIMESTAMP)"
"CREATE INDEX index_bands_id ON bands(id)"
"CREATE TABLE bands_members(id SERIAL PRIMARY KEY,
band_id INTEGER REFERENCES bands (id),
member_id INTEGER REFERENCES members (id),
created_at timestamp NOT NULL default CURRENT_TIMESTAMP)"])
(defn down []
["DROP TABLE bands"]
["DROP TABLE bands_members"])
</pre>
We should be running the migrations when we drop and rebuild the database. Change your Makefile's "build-db" target like this:
<pre class="brush: xml, highlight: 6">
...
# Builds the DB by dropping and recreating it
build-db:
dropdb --if-exists --username $(USER) $(DBNAME) -h $(HOST) -p $(PORT)
createdb --username $(USER) $(DBNAME) -h $(HOST) -p $(PORT)
lein clj-sql-up migrate
</pre>
Now when you run <code>make build-db</code> in your terminal, you should see the database recreated by dropping it first, creating it and running the migration at the end.<br>
(<a href="https://github.com/adomokos/kashmir/commit/cfe053fbc712e7125b80a440f8578d02a5ea6b0b" target="_blank">Commit point</a>.)
</p>
<p>In the <a href="http://www.adomokos.com/2015/10/clojure-api-with-yesql-migrations-part2.html" target="_blank">next post</a> we'll add seed data to the database for testing purposes, we will also use the excellent <a href="https://github.com/krisajenkins/yesql" target="_blank">Yesql</a> library to communicate with the database.Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.comtag:blogger.com,1999:blog-5573355675806579511.post-24889279685326784842015-10-02T10:12:00.001-05:002015-10-02T10:12:22.145-05:00The $18 Web Design<p>
My first employer in Minneapolis charged 10s of thousands of dollars for a brochureware website that only promoted a small business in the early 2000s. It became harder and harder to acquire new business, and the price point eventually dropped significantly, but still, it was a pretty good business to be in.
</p><p>
Then in August 2011 Twitter open sourced <a href="http://getbootstrap.com/" target="_blank">bootstrap</a>, which turned the entire web to look the same. It was a huge leap in the right direction, but every other website or internal app looked very much alike.
</p><p>
More and more UI engineers cranked out "customized" bootstrap apps. Engineers familiar with bootstrap were able to modify and tweak the design. I even built my <a href="http://iridge.com/resume/" target="_blank">own resume</a> on one of those bootstrap designs.
</p><p>
A couple of weeks ago a good friend of mine pinged me about help with his app prototype. He even picked out the <a href="https://wrapbootstrap.com/theme/inspinia-responsive-admin-theme-WB0R5L90S" target="_blank">template</a> for himself. He sent it over to me as a zip file, I extracted it and my jaw dropped. The 566 MB of content I found in it was amazing.
</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYx6V7ALx9ncUj_kCev2vdB18mFVb9wTBdbGY6cCVMoVJaI_cwSbVU1ZY3Ef_NMdyXyX00FLK_oIO2FJVWekdonjd83-IMn1U_6N1_rEmDfdQVnnH5u6jdlICts0PFZPysQOsPhz8MGx2T/s1600/inspinia.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYx6V7ALx9ncUj_kCev2vdB18mFVb9wTBdbGY6cCVMoVJaI_cwSbVU1ZY3Ef_NMdyXyX00FLK_oIO2FJVWekdonjd83-IMn1U_6N1_rEmDfdQVnnH5u6jdlICts0PFZPysQOsPhz8MGx2T/s450/inspinia.png" /></a></div>
<p>
It had:
<ul>
<li>4 layouts</li>
<li>5 dashboard mockups with sample graphs</li>
<li>7 different graphs</li>
<li>an email template with all messages, view and write email templates</li>
<li>metrics dashboard with 6 different data representations</li>
<li>all sorts of UI widgets like zoomable maps</li>
<li>form widgets with wizards, file uploads</li>
<li>19 user profile views</li>
<li>2 login, forgot password and different error pages</li>
<li>a code editor, a timeline view, tree and chat view</li>
<li>different UI elements like panels, buttons, tabs and badges</li>
<li>4 data table templates</li>
<li>full e-commerce design for products, orders, order detail and payment forms</li>
<li>3 galleries</li>
<li>2 menu options</li>
<li>a public site design, this way your company can have a marketing site aligned with the app</li>
<li>5 full and starter templates (angular.js, ASP.NET MVC, meteor.js, static HTML 5 template, Rails 4 templates)</li>
<li>3 CSS pre-processors (SASS, SCSS, LESS)</li>
<li>all the templates in PSD in case you want to further tweak it yourself</li>
</ul>
</p><p>
With a template like that you can pretty much do everything you want. An e-commerce app? Sure! A data analytics application? Absolutely. I've tried using my UI chops before, but none of my attempts were remotely close to what a template like this can offer.
</p><p>
And the best part? This template does not cost $1000. Not even $500. You can have it all for 18 dollars. Yes, for the cost of your lunch you can put your ideas in motion.
</p>Attila Domokoshttp://www.blogger.com/profile/09067995287578229487noreply@blogger.com