Thursday, January 15, 2015

The Case For and Against Cucumber

The TL;DR version

Cucumber has 3 benefits:

  1. Feature Discovery
  2. Automated Acceptance Testing
  3. (Executable) Documentation
In order to use Cucumber successfully within your organization, you need to take advantage of at least 2 of these benefits.

The Case For and Against Cucumber

Last week I gave a talk on Cucumber at CodeMash. I was glad to see the roughly 40 people who came to hear me despite being scheduled as one of the last sessions of the conference.

I ended my talk with this very personal story. I had worked in the Microsoft .NET space for 8 years, but I wanted to do something else. I was fascinated by the Ruby community, the innovation, the sharing I had seen among its members. I lived in Cleveland, OH, and there were only a handful of companies working with Ruby at that time.

My ticket to the Ruby World was my familiarity with Cucumber. My good friend - Joe Fiorini - approached me if I'd be interested in joining their company as a QA Engineer, helping them with QA automation with Cucumber. I was eager to say yes and joined them shortly after.

I wrote the first couple of features, showed them how to write Gherkin. Our test suite happily grew during the first few months of my employment. However, as more and more engineers joined, the case against Cucumber increased. Some of the engineers said they are not against acceptance testing, but those acceptance tests should be written in RSpec and not in Cucumber. Cucumber seemed to them an unnecessary extra layer they did not need.

I felt sad and disappointed. Why my fellow engineers were not seeing the value of Cucumber? What did I do wrong? Should I have spent more time explaining the values of executable documentation? I felt helpless. I asked Jeff "Cheezy" Morgan - who knows a lot more about the values and application of Cucumber at various organizations - to have breakfast with me and one of the engineers.

We met with Cheezy a few weeks later. I told him: "Cheezy, I think Cucumber is a fantastic tool, it expresses business logic like nothing else. Our company should use it. Please, be the judge here, what are we doing wrong?" Cheezy had one question: "Who is reading your Gherkin?" I said: "Who? It's us, the engineers, and maybe our QA folks." He said: "You should not use Cucumber, you would be better off with just RSpec. Cucumber is a tool for discovering requirements." "Huh?!"

I went back to work feeling a bit disappointed. I used Cucumber for acceptance testing, I did not want to hear about any other tools to do that.

It took me a few months to realize that Cheezy was right. I blindly used Cucumber for its expressiveness, and not for its value as a feature discovery tool.

Fast forward a few years to today and I wonder, why Cucumber or Gherkin is useful to us at Hireology. The answer is clear now: the entire Product, QA and Engineering team values and leverages Cucumber for feature discovery. Product will try writing a couple of scenarios when they brainstorm on a new feature. Those scenarios will be fine-tuned, extended with new ones during our 3 Amigos Meeting (a meeting to flush out feature requirements with Product, QA and Engineering). We just happen to automate those specifications during the development process.

I love how we start thinking about edge-cases well before the development begins with the help of Cucumber and Gherkin. What if the external system is not responding? Where will the user be redirected after a successful form submission? The benefit of doing this kind of planning is a more accurate estimation. Estimating engineering effort of a feature is hard, but if you know what you need to build, then at least you can take a decent stab at it, it won't be a complete swag.

We successfully use Cucumber for (1.) feature discovery and for (2.) automated acceptance testing. Now on to its third benefit: documentation.

Our Cucumber (Gherkin) scenarios are living together with our code base. Looking at them is still hard and not available for everyone at our company. I'd like to make all our features accessible to everybody, from our CEO to all our sales folks. "How does feature X work?" "I don't know, go through the feature document by clicking on this hyperlink."

Have you tried reading up on RSpec's mocking and stubbing functionality. In case you have, I am sure you have visited the Relish app. Take a look at the page that describes a basic functionality of RSpec mocking. Does it look familiar? Well, there is a Given/When/Then text in there. The most important question: is that useful? Can you learn the tool just by reading through that? That text is coming from RSpec's own source code. The RSpec developers packaged up their Cucumber scenarios and presented it in an elegant, nicely formatted, searchable app. Relish app is the prime example of executable documentation.

Publishing our more than 200 scenarios is my next goal. We use Cucumber for feature discovery, automated acceptance testing, we should use it for documentation as well.

Thursday, January 1, 2015

Fast TDD in Clojure with Fireplace.vim

I've been looking at Clojure for the past 18 months. I prefer learning and practicing by writing a failing test first, but unfortunately, the feedback loop through TDD in Clojure is slow. How slow? Well, it's between 4-10 seconds depending on the size of the project. I am still new to the language, I want to tweak my code a tiny bit and see if that change broke my tests. I am used to Ruby's almost immediate test execution, and the long wait for running the tests in Clojure makes it less effective.

In Ruby Land, I am used to running a large number of tests (958 examples in our application) in about 3.8 seconds. In a brand new Clojure project, it takes about 4 seconds to run the only failing test. This is no surprise: Clojure code has to be be compiled to Java byte code, where the compilation takes time.

I bumped into Ben Orenstein's great "Tips for Clojure Beginners" blog post a few weeks ago. It's a must read if you're new to Clojure. Vim is my preferred editor, and he wrote about a vim plugin by Tim Pope, called fireplace.vim. I remember looking at it briefly, but for some reason, I did not give it a try at that time.

A few days later I hacked on some code in Clojure again, and it reached a point where I threw my hands in the air and declared: "enough is enough!" I caught myself checking out Twitter and other websites as I had to wait about 10 seconds to run the tests after a simple change. I went through this blog post, where the author talks about using fireplace.vim for test execution. I gave it a try, and there is no turning back!

I installed fireplace.vim with pathogen. I opened another tab in my terminal, navigated to the root directory of my Clojure project. Fired up lein repl there and noted what the port number was.

In this case, 53844 was the port number for the nREPL server. I connected to that from my vim session in the other terminal tab by typing the vim command :Connect.

Fireplace gently investigated which nREPL server I wanted to connect to. I chose (the obvious) option one, it used localhost and I had to provide the port number from the other tab, which was 53844.

I submitted this option, and I was connected to the nREPL in the other tab. Fireplace lets me run the tests in the currently selected pane by using the :RunTests command. I did that, and much to my surprise the tests executed almost instantaneously. I did it once more (or maybe 5 times) just for the heck of it! This is what I found in the quickfix list:

I made the test pass, the output was terse. I guess there isn't much to say when all my expectations are passing. I included an animated gif here to show you what it feels like running the tests. Super sweet, don't you think!?

When I change a Clojure file in a different buffer (other than the buffer where my tests are), I need to Require! those files again. I get around this by writing all my functions right above the tests in the same buffer, and moving them to their final place when I feel confident about them.

There is an easier way to connect to a REPL by using fireplace.vim's :Piggieback! command. Please read the docs of this great vim plugin, that's how you can learn all the other features (like macroexpand) I have not described in this blog post.

My personal shortcut to run the tests is ,r. Setting it up with vim was easy:
:nmap ,r :RunTests<CR>. With this change, I had the same joy in Clojure as I've had with Ruby and RSpec for years. Bye-bye checking out while I am test driving my code in Clojure!

Saturday, December 6, 2014

Cucumber Ain't Bad, But You Might Be Doing It Wrong

My talk got selected for CodeMash 2015, and I was invited to speak there in January 2015. The best way to prepare for the talk is by practicing it a couple of times before the conference. I've worked hard on my message, on my slides, I even watched Ben Orenstein's presentation on "How to Talk to Developers".

This past Tuesday I gave the talk's initial version at the Chicago Ruby User Group. A good crowd came out to listen to what I had to say about the topic despite the cold, wintery weather. I received a couple of questions, had good conversations with some of the folks afterwards.

Are you unsure about the topic? Here is the pitch for my talk:

The cry from developers is unanimous: “I don’t like Cucumber. My acceptance tests are brittle and hard to maintain!” And in fact, they can be.

This presentation will show you what tools and techniques you should use to write independent, stateless step definitions that can be weaved together to form a testing DSL.

You can review the code examples in this repo.

And here are the slides:

Thursday, August 21, 2014

Unobtrusive Logging with a Decorator

I've recently added logging to LightService. The organizers pipe log messages to whatever you define as your logger.

I, [LS] - calling organizer TestDoubles::MakesTeaAndCappuccino
I, [LS] - @keys in context: :tea, :milk, :coffee
I, [LS] - executing TestDoubles::MakesTeaWithMilkAction
I, [LS -   expects: :tea, :milk
I, [LS] -   promises: :milk_tea
I, [LS] -     keys in context: :tea, :milk, :coffee, :milk_tea
I, [LS] - executing TestDoubles::MakesLatteAction
I, [LS] -   expects: :coffee, :milk
I, [LS] -   promises: :latte
I, [LS] -     keys in context: :tea, :milk, :coffee, :milk_tea, :latte

This blog post tells you the story of how I added logging to this gem I maintain.

What do you think of this code?

class WithReducer
  attr_reader :context

  def with(data = {})
    @context = LightService::Context.make(data)
    self
  end

  def reduce(*actions)
    raise "No action(s) were provided" if actions.empty?
    actions.flatten!

    actions.reduce(context) do |context, action|
      action.execute(context)
    end
  end
end

It's not that hard to figure out what's going on in this object. It has 2 methods, #with and #reduce. The #with method is simple, it creates a Context object through the factory method and returns self for chaining. #reduce takes an arbitrary number of actions and reduces them by calling their #execute method.

Ok, let’s add logging around this:

class WithReducer
  attr_reader :context

  def with(data = {})
    logger.info("[LS] - calling organizer <#{organizer.to_s}>")

    @context = LightService::Context.make(data)

    logger.info("[LS] - keys in context: #{extract_keys(decorated.context.keys)}")

    self
  end

  def reduce(*actions)
    raise "No action(s) were provided" if actions.empty?
    actions.flatten!

    actions.reduce(context) do |context, action|
      result = action.execute(context)

      logger.info("[LS] - executing <#{action.to_s}>")
      if defined? action.expects and action.expects.any?
        logger.info("[LS] - expects: #{extract_keys(action.expects)}")
      end
      if defined? action.promises and action.promises.any?
        logger.info("[LS] - promises: #{extract_keys(action.promises)}")
      end
      logger.info("[LS] - keys in context: #{extract_keys(context.keys)}")

      result
    end
  end

  private

  def extract_keys(keys)
    keys.map {|key| ":#{key}" }.join(', ')
  end
end

This is the exact same core logic as the one in the first example, except it got polluted with logging. This code works, but seeing what the two instance methods do is not obvious, far from simple. What's wrong here? First, two separate concerns (logging and the core functionality) got squashed into one. Second, when the logging logic changes the underlying core logic will be touched violating the Single Responsibility Principle.

Logging and the core functionality of this code should be separated out. The logging needs to wrap around (decorate) the core functionality. It should look something like this:

A decorator - the WithReducerLogDecorator in this case - publishes the exact same interface as the object it decorates, it has the with and reduce methods. I started with those when I went about refactoring this:

class WithReducerLogDecorator
  def with(data = {})
  end

  def reduce(*actions)
  end
end

Then I "wrapped" the original logic with this decorator. The log decorator delegates the calls to the wrapped object, which is WithReducer in this case:

class WithReducerLogDecorator
  attr_reader :decorated

  def initialize(decorated = WithReducer.new)
    @decorated = decorated
  end

  def with(data = {})
    decorated.with(data)
  end

  def reduce(*actions)
    decorated.reduce(*actions)
  end
end

At this point it does not matter which WithReducer class I'd use. Using the original one or the decorated should produce the exact same functionality.
I started adding logging around the decorated methods, refactoring the #with method was first:

class WithReducerLogDecorator
  attr_reader :decorated, :organizer

  def initialize(decorated = WithReducer.new, organizer)
    @decorated, @organizer = decorated, organizer
  end

  def with(data = {})
    logger.info("[LS] - calling organizer <#{organizer.to_s}>")

    decorated.with(data)

    logger.info("[LS] - keys in context: #{extract_keys(decorated.context.keys)}")
    self
  end

  ...
end

Adding logging around the #reduce method was a bit harder. It reduces the provided actions by calling their execute method. There is no easy way to hook into that process just yet. However, calling the provided block within the reduce block opens up the routine for extension (Open/closed principle). I changed the original reduce method like this:

class WithReducer
  ...

  def reduce(*actions)
    raise "No action(s) were provided" if actions.empty?
    actions.flatten!

    actions.reduce(context) do |context, action|
      result = action.execute(context)
      # ::: Invoke the provided block :::
      yield(context, action) if block_given?
      result
    end
  end
end

The logging logic can now wrap around the #reduce method like this:

class WithReducerLogDecorator
  ...

  def reduce(*actions)
    decorated.reduce(*actions) do |context, action|
      # ::: All this code is executed within the decorated reduce block :::
      logger.info("[LS] - executing <#{action.to_s}>")
      if defined? action.expects and action.expects.any?
        logger.info("[LS] - expects: #{extract_keys(action.expects)}")
      end
      if defined? action.promises and action.promises.any?
        logger.info("[LS] - promises: #{extract_keys(action.promises)}")
      end
      logger.info("[LS] - keys in context: #{extract_keys(context.keys)}")
    end
  end

  private

  def extract_keys(keys)
    keys.map {|key| ":#{key}" }.join(', ')
  end
end

And with that I wrapped the original #with and #reduce methods with logging from outside with a decorator. I separated the two concerns: the core logic and the logging around it. I only need to touch the decorator object when the logging logic needs to be changed and I don't even need to look at the logging code when the core logic changes.

I learned about these concepts 9 years ago when I used Aspect Oriented Programming (AOP). I was fascinated by Around Advices, where logging was applied around a routine, beautifully separating the concerns.

Saturday, July 19, 2014

Use GitHub as Your Professional Portfolio

I've been doing quite a bit of candidate interviewing for the "Ruby Engineer - Remote" position we would like to fill. Our thought-out process starts with filtering out applicants based on their resume, their cover letter and their digital portfolio (website, blog or GitHub profile). I am surprised how many people have a GitHub account with no activity at all. I think that's wrong.

I took a look at my own GitHub profile. The activity looks pretty nice when I include the private projects.

However, when I looked at it as a non-authenticated user, the picture is not so stellar.

I would be happy to see candidates with as much public activity as I have. Do I contribute to open source software? Sometimes, not as often as I'd like. Then how do I have this much activity? Simple: I practice and I push everything I can to GitHub.

I want to advertise what languages I am playing with, what tools and frameworks I've been looking at. When I read books I create a quick and simple repository, I practice the examples by writing a failing test against them and when I reach the end of a chapter I commit and push the code to GitHub.

My GitHub profile is my own professional practicing-learning diary. I've recently started to look into Erlang and Clojure. Am I any good at these languages? No, or not yet! As I read books, learn new things and practice I'll sure get better. Last fall I looked at Erlang, and in November I switched to Clojure. By January I worked on the fizz-buzz kata and I got as far as playing with a Tic-Tac-Toe game in Clojure. Then I started to focus on my own gem, LightService and these days I am back at looking at Clojure.

As a professional, you've been learning a new language, you've been toying with the latest JavaScript MVC frameworks, why don't you broadcast that information? Don't let it sit on your computer! More and more companies will check your GitHub activity as you try to find a new job. They would be happy to see you're curious, you've been learning and you care about your own professional future.

Do yourself a favor and push as much of your learning to GitHub as possible. People - like me - looking at your profile there will jump on candidates who are "radiating that information". And it does not have to be GitHub, it can be BitBucket or anything else.

Just make sure whatever you do, it gets out there!

Wednesday, May 14, 2014

Expects and Promises in LightService

I received an issue a few weeks ago for the LightService project:

@adomokos, I saw an internal tool used at Github that is a lot like LightService. One nice they had was this notion of Promises and Expectations. Used the following way:

class FrothMilk
  include LightService::Action
  expects :milk, :cup
  promises :frothed_milk_in_cup

  executed {...}
end

You can immediately tell what this action expects from the context, and what this action promises to set in the context. An organizer would find this useful because it can see the chain of actions and check if they're compatible by looking at the expectations and promises.
What do you think? Would you like this included in LightService?

Oh wow! Github is using something similar to LightService? I'd like to see that! The concept of expects and promises sounded like a very good idea, I asked the person to submit a Pull Request for it. I found it pretty good when it arrived a few weeks later, but I wanted to polish it up a bit more.

This is how I had been writing actions with LightService:

class FooAction
  include LightService::Action

  executed do |context|
    baz = context.fetch :baz

    bar = baz + 2
    context[:bar] = bar
  end
end

When I look at this action - and many others I have written over the years - I notice a common pattern: I have to pull items from the context I want to work with, and I need to push the items I need to expose back into it. Although I try to keep actions small, doing this is repeated all over.

I prefer to call fetch when I pull an item from the context hash as the call throws a KeyError when the hash does not have the key I am asking for. Using the accessor baz = context[:baz] would force me dancing around nils in the code which I try to avoid.

The expects macro will help you no matter how you access the value in the hash as the LightService action will throw an ExpectedKeysNotInContextError if the key is not found in the context. The action expects the key to be in the context and when it's not provided, the call will blow up right there, no further processing will take place.

The action can be changed this way after the expects macro is introduced:

class FooAction
  include LightService::Action
  expects :baz

  executed do |context|
    baz = context.fetch :baz

    bar = baz + 2
    context[:bar] = bar
  end
end

Having the expects macro solved the problem of making sure the key is in the context before execution. However, I wanted to see if this macro could be used to generate accessors in the Context on the fly with the given key. This way I don't need to pull the items from the context one by one, I could just call this "magic" reader that fetches the value for me. With a bit of metaprogramming I was able to do just that. The action can be changed like this:

class FooAction
  include LightService::Action
  expects :baz

  executed do |context|
    # Notice how I use `context.baz` here
    bar = context.baz + 2
    context[:bar] = bar
  end
end

This is all great, but what about promising what the context hash will have after execution? The promises macro does that; when the promised key is not in the context, a PromisedKeysNotInContextError is thrown. This is how the action looks after using the promises keys:

class FooAction
  include LightService::Action
  expects :baz
  promises :bar

  executed do |context|
    # Notice how I use `context.baz` here
    bar = context.baz + 2
    context[:bar] = bar
  end
end

Setting the value of "bar" under the :bar keyword in the context is repetitive. I tried to see if I could do something similar to the expects macro and use it to set value in the context hash besides verifying the key is there. Since blocks in Ruby have well defined closures, peeping in and setting values are not feasible. I chose Context accessors with the promised keys, this way when a value is being set through these, LightService will put it in the context for you.
The above example can be further simplified:

class FooAction
  include LightService::Action
  expects :baz
  promises :bar

  executed do |context|
    context.bar = context.baz + 2
  end
end

Now look how we went from dealing with items in the hash explicitly to focusing on the logic itself:

class FooAction
  include LightService::Action
  expects :baz
  promises :bar

  executed do |context|
    # No need to fetch item from the context
    # baz = context.fetch(:baz)

    context.bar = context.baz + 2

    # No need to add the item to the context
    # context[:bar] = bar
  end
end

Feel free to follow the same refactoring in this spec.

You could argue that this forces you to use a DSL and it's not saving a line of code for you. I hear you, but keep in mind, the expects and promises macro does a whole lot more work than just pulling and putting items in and out of the context: it provides an input/output contract for the action.

With the knowledge of what the inputs and outputs of actions are, the time when I can verify the keys in the Organizer pipeline is not far. I should be able to ask the Organizer to verify the order of actions, as actions might need a product from an action that was executed earlier in the Organizer dictated pipeline. Dependency check FTW!

Sunday, April 27, 2014

Tracking Progress of a Rails Rewrite

I inherited a challenging codebase about 14 months ago. The code did not have tests, there were methods with 100-200 lines of code in them filled with conditionals and iterators. It was obvious: we needed a rewrite. Ok, we were not thinking about The Big Rewrite, but we knew the code had to be cleaned up. I had all the support from the leadership team, but I was asked for one thing: provide a metric to track progress.

A few months went by and I did not have an answer. I couldn’t tell how fast or slow we can get done with the cleanup. Is it going to be six months, one year, maybe two? I did not know what to say.

My goal was extracting business logic from Rails controllers and models and putting them into lib/services. I wanted to structure the code into stateless, reusable actions with LightService, decoupling it from Rails as much as possible. I trusted these services as test coverage for them was fairly high. I knew no matter what business logic I had to implement, the small service classes were able to take anything I would throw at them.

The app used - the now unsupported - Data Mapper library that we wanted to get out of and transition over to Active Record. We put all our AR classes in the app/models/ar directory under the "AR" namespace. We kept these models clean, well tested. Now we trusted everything under lib/services and app/models/ar.

We also started to extract logic into RESTful controllers that had very little logic in them. These controllers were only a handful, maybe half a dozen. I realized we did not do much action clean up in the existing controllers. In fact, we shifted the responsibility to new controllers and services that we trusted.

One of my favorite Ruby code analysis tool is flog. Flog gives you complexity points based on "ABC" metrics: assignments, branches and conditionals. You can run flog against your entire controller and model code and you get a complexity point. If you don't have much logic in the views or JavaScript, that number is your application's complexity.

Comparing trusted and untrusted total lines of code can provide a number, but I am not sure how much I could trust that. 10 lines of really crappy code loaded with iterators and conditionals compared to 10 lines of clean, readable and tested code just does not provide a one-to-one ratio. Why not looking at this problem from the complexity side? 5 point flog complexity can be on 1 single line in a crappy code, but it can be on 4 different lines in a method in the cleaned up code. Amount of coded logic should be expressed in complexity points and not in the number of lines of code.

I realized I could actually trust the flog complexity points to compare the trusted and untrusted code ratio. I easily calculated the total controller complexity. By subtracting the trusted controller complexity from the total complexity I had the untrusted complexity for controllers. I did the same for models. I put all the complexity from lib/services under the trusted bucket. Deviding the total trusted complexity by the untrusted code complexity provided the trusted/untrusted code ratio.

Here is an example of how the calculation worked:

Total controller complexity:8945
Trusted controller complexity:489
Untrusted controller complexity (8945 - 489):8456
 
Total model complexity:1498
Trusted model complexity:249
Untrusted model complexity (1498 - 249):1249
 
Trusted services complexity:845
 
Untrusted total complexity:9705
Trusted total complexity:1583
 
Trusted/Untrusted code ratio:1583/9705 * 100 = 16.3%

The numbers in the example above are made up, they don't reflect the code of my current employer.

A month later you can do the same calculation and you’ll see that the Trusted/Untrusted code ratio is 19.5%. Well, look at that, 3% of your code just gained trust! That new 3% of the code is easy to change as it's small, passes all the tests, communicates intent and has no duplication.

By yielding 3% more trusted code in a month, you will need more than 2 years to clean up the existing code base unless you can accelerate the code clean up somehow.