Thursday, January 15, 2015

The Case For and Against Cucumber

The TL;DR version

Cucumber has 3 benefits:

  1. Feature Discovery
  2. Automated Acceptance Testing
  3. (Executable) Documentation
In order to use Cucumber successfully within your organization, you need to take advantage of at least 2 of these benefits.

The Case For and Against Cucumber

Last week I gave a talk on Cucumber at CodeMash. I was glad to see the roughly 40 people who came to hear me despite being scheduled as one of the last sessions of the conference.

I ended my talk with this very personal story. I had worked in the Microsoft .NET space for 8 years, but I wanted to do something else. I was fascinated by the Ruby community, the innovation, the sharing I had seen among its members. I lived in Cleveland, OH, and there were only a handful of companies working with Ruby at that time.

My ticket to the Ruby World was my familiarity with Cucumber. My good friend - Joe Fiorini - approached me if I'd be interested in joining their company as a QA Engineer, helping them with QA automation with Cucumber. I was eager to say yes and joined them shortly after.

I wrote the first couple of features, showed them how to write Gherkin. Our test suite happily grew during the first few months of my employment. However, as more and more engineers joined, the case against Cucumber increased. Some of the engineers said they are not against acceptance testing, but those acceptance tests should be written in RSpec and not in Cucumber. Cucumber seemed to them an unnecessary extra layer they did not need.

I felt sad and disappointed. Why my fellow engineers were not seeing the value of Cucumber? What did I do wrong? Should I have spent more time explaining the values of executable documentation? I felt helpless. I asked Jeff "Cheezy" Morgan - who knows a lot more about the values and application of Cucumber at various organizations - to have breakfast with me and one of the engineers.

We met with Cheezy a few weeks later. I told him: "Cheezy, I think Cucumber is a fantastic tool, it expresses business logic like nothing else. Our company should use it. Please, be the judge here, what are we doing wrong?" Cheezy had one question: "Who is reading your Gherkin?" I said: "Who? It's us, the engineers, and maybe our QA folks." He said: "You should not use Cucumber, you would be better off with just RSpec. Cucumber is a tool for discovering requirements." "Huh?!"

I went back to work feeling a bit disappointed. I used Cucumber for acceptance testing, I did not want to hear about any other tools to do that.

It took me a few months to realize that Cheezy was right. I blindly used Cucumber for its expressiveness, and not for its value as a feature discovery tool.

Fast forward a few years to today and I wonder, why Cucumber or Gherkin is useful to us at Hireology. The answer is clear now: the entire Product, QA and Engineering team values and leverages Cucumber for feature discovery. Product will try writing a couple of scenarios when they brainstorm on a new feature. Those scenarios will be fine-tuned, extended with new ones during our 3 Amigos Meeting (a meeting to flush out feature requirements with Product, QA and Engineering). We just happen to automate those specifications during the development process.

I love how we start thinking about edge-cases well before the development begins with the help of Cucumber and Gherkin. What if the external system is not responding? Where will the user be redirected after a successful form submission? The benefit of doing this kind of planning is a more accurate estimation. Estimating engineering effort of a feature is hard, but if you know what you need to build, then at least you can take a decent stab at it, it won't be a complete swag.

We successfully use Cucumber for (1.) feature discovery and for (2.) automated acceptance testing. Now on to its third benefit: documentation.

Our Cucumber (Gherkin) scenarios are living together with our code base. Looking at them is still hard and not available for everyone at our company. I'd like to make all our features accessible to everybody, from our CEO to all our sales folks. "How does feature X work?" "I don't know, go through the feature document by clicking on this hyperlink."

Have you tried reading up on RSpec's mocking and stubbing functionality. In case you have, I am sure you have visited the Relish app. Take a look at the page that describes a basic functionality of RSpec mocking. Does it look familiar? Well, there is a Given/When/Then text in there. The most important question: is that useful? Can you learn the tool just by reading through that? That text is coming from RSpec's own source code. The RSpec developers packaged up their Cucumber scenarios and presented it in an elegant, nicely formatted, searchable app. Relish app is the prime example of executable documentation.

Publishing our more than 200 scenarios is my next goal. We use Cucumber for feature discovery, automated acceptance testing, we should use it for documentation as well.

Thursday, January 1, 2015

Fast TDD in Clojure with Fireplace.vim

I've been looking at Clojure for the past 18 months. I prefer learning and practicing by writing a failing test first, but unfortunately, the feedback loop through TDD in Clojure is slow. How slow? Well, it's between 4-10 seconds depending on the size of the project. I am still new to the language, I want to tweak my code a tiny bit and see if that change broke my tests. I am used to Ruby's almost immediate test execution, and the long wait for running the tests in Clojure makes it less effective.

In Ruby Land, I am used to running a large number of tests (958 examples in our application) in about 3.8 seconds. In a brand new Clojure project, it takes about 4 seconds to run the only failing test. This is no surprise: Clojure code has to be be compiled to Java byte code, where the compilation takes time.

I bumped into Ben Orenstein's great "Tips for Clojure Beginners" blog post a few weeks ago. It's a must read if you're new to Clojure. Vim is my preferred editor, and he wrote about a vim plugin by Tim Pope, called fireplace.vim. I remember looking at it briefly, but for some reason, I did not give it a try at that time.

A few days later I hacked on some code in Clojure again, and it reached a point where I threw my hands in the air and declared: "enough is enough!" I caught myself checking out Twitter and other websites as I had to wait about 10 seconds to run the tests after a simple change. I went through this blog post, where the author talks about using fireplace.vim for test execution. I gave it a try, and there is no turning back!

I installed fireplace.vim with pathogen. I opened another tab in my terminal, navigated to the root directory of my Clojure project. Fired up lein repl there and noted what the port number was.

In this case, 53844 was the port number for the nREPL server. I connected to that from my vim session in the other terminal tab by typing the vim command :Connect.

Fireplace gently investigated which nREPL server I wanted to connect to. I chose (the obvious) option one, it used localhost and I had to provide the port number from the other tab, which was 53844.

I submitted this option, and I was connected to the nREPL in the other tab. Fireplace lets me run the tests in the currently selected pane by using the :RunTests command. I did that, and much to my surprise the tests executed almost instantaneously. I did it once more (or maybe 5 times) just for the heck of it! This is what I found in the quickfix list:

I made the test pass, the output was terse. I guess there isn't much to say when all my expectations are passing. I included an animated gif here to show you what it feels like running the tests. Super sweet, don't you think!?

When I change a Clojure file in a different buffer (other than the buffer where my tests are), I need to Require! those files again. I get around this by writing all my functions right above the tests in the same buffer, and moving them to their final place when I feel confident about them.

There is an easier way to connect to a REPL by using fireplace.vim's :Piggieback! command. Please read the docs of this great vim plugin, that's how you can learn all the other features (like macroexpand) I have not described in this blog post.

My personal shortcut to run the tests is ,r. Setting it up with vim was easy:
:nmap ,r :RunTests<CR>. With this change, I had the same joy in Clojure as I've had with Ruby and RSpec for years. Bye-bye checking out while I am test driving my code in Clojure!

Update on 01/31/2015

I've been using this keybinding with fireplace in vim recently: :nmap ,r :Require! <bar> Eval (clojure.test/run-tests)<CR>. It picks up any changes I make in the source and the test files as I require packages before every test run. I'd recommend giving this a try.