A Tale of Two Rails Rewrites

The TL;DR Version

Watch the video "So You Think You Need a Rewrite" by Chad Fowler and Rich Kilmer. Kill your old app with a Strangler App as you always, constantly release new code.

Preface

I worked for a software company that tried to replace its aging software with a new one. Incremental releases were not possible, it was an attempt to do a "Big Bang" rewrite. The customers received the preview 5 years after development started and due to the lack of smooth upgrade nobody was willing to take the brand new software. The economy turned sour, people were laid off and tens of millions of dollars later - after I left the company - the project was canned.

JRubyConf in 2010

I attended JRubyConf in Columbus, Ohio in 2010 and watched the talk "So You Think You Need a Rewrite" by Chad Fowler and Rich Kilmer. I had read Chad's book "The Passionate Programmer" before I heard his talk there. Little I knew about rewriting Rails applications at that time, but the main message of their talk sticked with me for a long time: "Never, ever do a big bang rewrite!"

The First

Both of the startups I worked for recently had two things in common: the codebase was fairly old (3-5 years) and neither of them had automated tests when I started on the project.

I knew rewriting the entire app was not possible. It would have taken too long, it would have been too expensive, and we wouldn't have been able to release anything to production until the project was done. What we attempted was an "in-place", partial rewrite. Instead of taking the risk of rewriting the entire app we decided to do it in piecemeal: rewriting the app one product at the time.

We kept the database and the Active Record ORM layer intact and rewrote everything above it just for that product. This meant about 15% of the entire app. The new app lived side by side to the old Rails app, the routing engine took the users to the old or to the new product depending on which one they were set up with.

This approach was less risky, but we could not deliver the project until the entire product was rewritten in the app even if it meant just 15% of the codebase. It was still a minified all-or-nothing approach where the code finally made it into production, but deadlines were missed a couple of times.

The Second

We have a 2 years old Rails app with about 17k LOC at my current company - Hireology. It has served its purpose and we are now at a point where we are ready to "refactor" the app. The question is how? We have to face several constraints:

• We only have two engineers
• We have to develop new features and enhancements
• We have to deploy our code several times a week
• We constantly have to increase the code quality

I read about the "Strangler App" idiom by Martin Fowler years ago. I really loved this concept: you start with a new app, as it grows it has more and more of the business logic and functionality of the old app and it takes over everything in the end, slowly but surely strangling the old app.

Progressive Insurance built their Claims application this way. They wanted to get off of the mainframe and rewrite the app with ASP.NET. The first thing they did was replacing the outdated black and green terminal UI with similarly looking web page talking to the mainframe. This change went smoothly for the employees who used this application since they barely noticed they were using a web page instead of a terminal screen. The same key bindings worked and it looked just like the terminal app.

Then every two or four weeks a tiny new feature went into production. First just a select box with country codes came from the app server, everything else was fed by the mainframe. Then a form was replaced by .NET web forms. And this went on for a year or more and eventually the same web UI was talking entirely to app servers and SQL Server databases cutting off its reliance on the mainframe, slowly strangling it to "death".

We will try something similar. Since all our clients are asking for our APIs, exposing our business logic through them makes a lot of sense. The old app runs on an older version of JRuby and this brand new Rails API app will be using Rails 4 with Ruby 2.0. It will be properly tested, when a new version of Rails comes out this app should be running on it shortly after.

We start with a tiny feature: there is one page in the app where we have to load all the different job profiles a client has. The old app receives the request, its controller wraps the params and sends it over to this new API app. The API app is sitting on top of the exact same database as the old Rails app. It collects the open, closed and pending jobs, serializes the collections to JSON. Passes it back to the old app which loads up the models from the received JSON data. It renders the view with data it just received from the API app without talking to the database again.

This feature is small, the users will never notice we started on this project and we will be able to release it in a week or two. If we keep doing this, moving the business logic into our new Rails API app, little by little we will shrink the legacy Rails app to just a dumb presentation engine. I am pretty sure that will be further reduced by building a rich HTML app - using a client side MVC or MVVM framework like Ember.js or Spine.

I was asked how long this will take. Maybe a year. Maybe less, maybe more. It's hard to say without having any kind of baseline data.
We will constantly move logic to our new API app where the code will have supporting automated tests. This API application - if it's structured properly - can feed not only our own app, but other third-party applications as well if they want to integrate our business logic into their own apps.

In the end, everybody wins.

Crossroads

The TL;DR Version

I like working for startups even if it's a gamble and I can lose my job fast. I enjoy the freedom it provides: I work from anywhere I want, I use the tools I like and I am working with passionate people.
I am moving to Chicago in the summer.

The Story

I wrote about frequent job changes and I was hoping my current job would improve the track record. It did not.

There were a couple of warning signs triggering my job search right after my CodeMash talk. I knew the market for Ruby jobs is very limited in Northeast Ohio and I remembered what it was like in Chicago. My wife was OK with moving to a bigger city so I started my search in the greater Chicago area early January.

First of all I wrote a resume. I don't believe in resumes, but I was targeting a geographic area where I have very few contacts. I pinged friends, tried to find out what's available through them. I also felt that time is running out and in an inexplicable moment, I uploaded my resume to Workbridge Associates' web site. Five minutes later my phone rang. We chatted for a few minutes and later that day I started receiving company profiles looking for Rails/Ruby developers. It was great, I found some amazing companies that I would have never found on my own.

I was on the phone quite a bit in the next two weeks. I talked to the headhunter firm and with the companies looking for developers. I must say having a blog and a Github repo helped me a lot. The hiring managers, developers were able to read up on how I think, what problems I solve and what kind of person I am. I was also happy to see that no company wanted to "grill me" on algorithms and different binary trees, they gave me homework assignments instead, which I think is a much better way of figuring out what the developer knows, how he/she thinks.

By the beginning of February my job search narrowed down to four different companies:

• Table XI
• Hireology
• Aggrego
• GiveForward

I took the Megabus to Chicago on the 3rd of February and by midnight I was in my hotel room anxiously waiting for the next morning to talk with these great companies.

Table XI gave me a homework assignment which I completed ahead of their deadline. I did not build a CLI interface to feed files into the program, I proved out it's behavior with Cucumber features. I also missed a scenario where you could order multiple times from the same dish.
Table XI invests a lot in their culture. I met with them three times while I was in Chicago. We went through my homework assignment, they had me code on the board and I also had to make changes to a Rails app I have not seen before. I believe they are very selective of who they hire, but again, their team seemed great. Oh, and they have their own chef who prepares lunch four times a week.

I met with the team of Hireology next. A printed paper was taped to their door saying "Hireology welcomes Attila Domokos!". I was flattered.

The company's CEO took me out for lunch. We chatted about our families, what I thought of their product while we ate our delicious soup and sandwich at a restaurant that did not look much from the outside.
After lunch the CEO, the head of product development and I sat down for an afternoon session where they talked about their business, about their growth potential and why they believe in their product. I showed them how I write software, what it means to run tests with and without Rails. But I think they were most intrigued by the executed Gherkin scenarios through Cucumber.
I got together with the CTO of their sister company Urban Bound to find out how I code with others.
At the end of the day we went down to the first floor and had a couple of beers at the pub. By the way, in Chicago almost every office building has a pub or a wine bar on the ground floor. You don't even have to leave the building!

The next day I visited Aggrego which is a small startup under a big company. They are working on the next generation of online news media for both news creation and consumption. Their team is extended with a couple of Thoughtworks engineers. I paired with Brian Kung in the morning, we added a few specs to cover his helper that loaded JS or static images for banner ads depending on the type of browser that made the request. I spent most of my time with Dave Willkomm in the afternoon, we test drove an EC2 server facade object they used in their deployment scripts.

The last day I took a quick cab ride to Giveforward, which is a startup in the Wicker Park neighborhood of Chicago. Their fundraising software helps with medical and other related expenses. Their application is written in PHP and they are ready to port it to Ruby on Rails. I met with two engineers and a designer/UX person. After their product demo and going through some their code I did a quick demo of how I build software. We did a quick brainstorming session on URL structures and on database entity relationships. I had a good conversation with them and their app would be interesting to work on.

After all this, which company did I chose?

Well, not so fast! I was about to get on the bus and travel back to Cleveland when I started receiving news from my coworkers about layoffs happening at my current employer. I knew it was coming, I just did not know it would happen so fast.

Before I answer the question I have to tell you: I liked all the companies I visited. None of them was perfect, there were things I liked in one and were missing in others.

However, I felt a special connection with the team at Hireology. I was amazed by how much they cared about their product, the amount of work they put into it and how firmly they believed in their success. I was also applying for the lead engineer position allowing me to define their development process, work out the details of the relationship between product development and engineering. Building an engineering team is something I found very attractive as well.

By the third and last day of my trip, after exchanging a couple of emails with Adam Robinson, the CEO of Hireology I knew I wanted to work with them. Two hours after I heard the layoffs that I was - unfortunately - part of and when I was riding the Megabus back to Cleveland, I received a fantastic offer that I could not refuse. I'll join Hireology today as their CTO, where I will not only oversee their software development, but I'll get to work on other areas that affect the success of the company.

I'll start my work remote, but I'll make frequent, one week long trips to Chicago so I can work with others in person at Hireology. I'll be back in Europe for seven weeks in May and June and we are moving to the North suburbs of Chicago in July.
I love Chicago, I fell in love with that city the first time I visited in 1998. Seeing the sparkling startup life there just proves it to me that my wife and I made a great decision.

So my dear friends in Northeast Ohio, let's spend time together while I am around. I'll try to attend user group meetings, coding events, work from coffee shops or from your office. I may be leaving soon, but I am sure I'll be back. Maybe for CodeMash next year?!

User Devised - Disconnecting Your User Entity From Devise

We have several Rails apps with duplicated user information. If you're John Smith in one of the apps, we will create a new user with the same information in the others. That's crazy, single sign-on should be the obvious solution.

In order to get there we had to disconnect the authentication mechanism from our User entity. Once we have the same separation in our apps, pulling the authentication out into a service or API should be very easy.
Also, I like to keep the User model application specific. I would much rather have fields that are relevant to our app needs in our User model and not mix devise's fields into that entity.
Here is what I had to do to disconnect devise from our User model.

First of all, I had to generate a member table with the following migration:

class AddMembers < ActiveRecord::Migration
  def change
    create_table(:members) do |t|
      ## Database authenticatable
      t.integer :user_id
      t.string :email,              :null => false, :default => ""
      t.string :encrypted_password, :null => false, :default => ""
      t.string :first_name,         :default => ""
      t.string :last_name,          :default => ""
      t.string :middle_name,        :default => ""

      ## Recoverable
      t.string   :reset_password_token
      t.datetime :reset_password_sent_at

      ## Rememberable
      t.datetime :remember_created_at

      ## Trackable
      t.integer  :sign_in_count, :default => 0
      t.datetime :current_sign_in_at
      t.datetime :last_sign_in_at
      t.string   :current_sign_in_ip
      t.string   :last_sign_in_ip

      ## Confirmable
      t.string   :confirmation_token
      t.datetime :confirmed_at
      t.datetime :confirmation_sent_at
      # Only if using reconfirmable
      t.string   :unconfirmed_email 

      ## Lockable
      # Only if lock strategy is :failed_attempts
      t.integer  :failed_attempts, :default => 0 
      t.string   :unlock_token
      # Only if unlock strategy is :email or :both
      t.datetime :locked_at

      ## Token authenticatable
      t.string :authentication_token

      # Uncomment below if timestamps were not included
      # in your original model.
      t.timestamps
    end

    add_index :members, :email,                :unique => true
    add_index :members, :reset_password_token, :unique => true
    add_index :members, :confirmation_token,   :unique => true
    add_index :members, :unlock_token,         :unique => true
    add_index :members, :authentication_token, :unique => true
  end

  def self.down
    # By default, we don't want to make any assumption
    # about how to roll back a migration when your
    # model already existed. Please edit below which fields
    # you would like to remove in this migration.
    raise ActiveRecord::IrreversibleMigration
  end
end

Instead of putting all the Member related code into the User model, I created a concern that can be mixed into the User model. This module can be reused in all of the Rails apps to connect the Member entity to the User.

require 'active_support/concern'
require 'active_support/core_ext/module'

module ModelExtensions
  module UserDevised
    extend ActiveSupport::Concern

    included do
      has_one :member
      validates_associated :member

      after_save :save_member, :if => lambda {|u| u.member }
      delegate :last_sign_in_at, :password, :password=,
               :password_confirmation, :password_confirmation=,
               :to => :member

      before_validation do
        self.member.first_name = self.first_name
        self.member.middle_name = self.middle_name
        self.member.last_name = self.last_name
        self.member.email = self.email_address
      end
    end

    def initialize(*params)
      super(*params)
      self.build_member(*params) if member.nil?
    end

    def save_member
      self.member.save!
    end
  end
end

I'd like to mention a couple of things about the module you see above. I am using one-to-one relationship between User and Member. Whenever I save a User, I save the Member as well by using the after_save callback. Admins can overwrite the users' password, I delegate those fields from User to Member, so the Member entity remains hidden behind the User. When a new User is initialized, I create a Member entity as well as you see it in the initialize method.

Devise strategies and overrides are defined in the Member model.

class Member < ActiveRecord::Base
  belongs_to :user

  # Include default devise modules. Others available are:
  # :token_authenticatable, :confirmable,
  # :lockable, :timeoutable and :omniauthable

  devise :database_authenticatable, :lockable, :timeoutable,
         :recoverable, :trackable, :validatable, :omniauthable
         # non-registerable :registerable, :rememberable

  validates :first_name, :last_name, :presence => true

  def active_for_authentication?
    # Comment out the below debug statement to view
    # the properties of the returned self model values.

    super && user.active?
  end

  def confirmed_account?
    (self.last_sign_in_at.nil? == false &&
        self.reset_password_token.nil?)
  end

end

I remember how hard it was to find information on the separation of the application specific User and the devise Member entity when I was working on this. I hope someone will find this code helpful, it would have helped me a lot.

Simple & Beautiful Rails Code with Functional Style

In early October I submitted this abstract to CodeMash:

"Do you have to look at Rails models with 1400 lines of code? Or 80 line method loaded with iterators, conditionals and instance variables? Not only you, even the code author does not understand what's going on in there.

I'll show you how you can craft simple and beautiful Rails application by adopting functional programming inspired ideas. Say goodbye to the mess you have by constructing tiny classes and functions that you can use to build up a complex system."

My talk got accepted and I gave it in early January, 2013. In case you could not attend it, or if you want to review the slides again, here is an abridged version of it.

npm and bundler - the Good Parts

I need two things from a language to take it seriously: good testing framework and some kind of package management. I've dealt with testing in my previous blog posts, now I compare two package management tools that I like, npm and bundler.

:: NPM

Early this year I went fairly deep into node.js development. I wanted to understand how it works and where I could use it in the future. Getting node on your local development environment is easy, but getting used to coding with async callbacks takes a bit of practice to master.

All I needed on OSX to get started with node was:

$: brew install node

When I wanted to use a package - let's say mocha.js - I just used npm.

$: npm install mocha

It installed everything into the project root under the node_modules directory, locally by default like this:

$: lltree
   some_nodejs_project
   |--node_modules
   |----mocha
   |------bin
   |------images
   |- ...

Sure you can have it installed globally, but you need to use the "-g" flag to do that. I did not want to pollute my node installation, I was glad with its default behavior where it installed everything under node_modules.

You do pay a price for this behavior. When you execute "mocha" in the terminal, the executable is not found.

% mocha
zsh: command not found: mocha

You have to locate that file under the node_modules directory. It's in "node_modules/mocha/bin" dir, by the way.
To get around this I just use a Makefile with a couple tasks in it:

REPORTER = list

test: test-bdd

test-bdd:
 @./node_modules/mocha/bin/mocha \
  --reporter $(REPORTER) \
  --ui bdd \
  -- spec/*.js

test-doc:
 @./node_modules/mocha/bin/mocha \
  --reporter $(REPORTER) \
  --ui bdd \

This way I can easily run my tests by executing "make" in the terminal.

:: BUNDLER

I switched from rvm to rbenv early this year. Rbenv with bundler makes a very powerful combo but when you install a gem it's going to install it globally by default.
Not good. I want to keep my gems local to the current project and I don't want to pollute my Ruby install with different versions of gems. What if I use Rails 3.2.9 in one project but I have to use 3.1.1 in another? Sure you could use rbenv-gemsets to get around this, but I already started using node with npm and I wanted to have a similar experience.

The "--path" switch in bundler lets me specify which directory I want to install my gems into. When I start a new project I immediately create a Gemfile. It's very simple, all you need is this:

source "http://rubygems.org"
gem "light_service", "->0.0.6"
gem "rspec"

Then I install all the gems through bundler with this command:

$: bundle install --path vendor/bundle --binstubs

Bundler puts all my gems under the vendor/bundle directory and creates a bin directory with executables for the gems that produce such a file. When I run rspec for my project this is what I do:

$: bin/rspec spec

You could either use "bin/rspec" or "bundle exec rspec", either works.

As you see, nor npm, neither bundler has the best solution. But they have facets that I like in both.

	default local install	easy access to executables
npm
bundler

Can we sync the good parts? Could both have local install by default with easy access to the executables?

Update

Shortly after I published this post, my good friend Joe Fiorini pinged me on twitter. Here is our conversation:

I did not know that npm creates a ".bin" directory under "node_modules" with symlinks pointing to the individual executable files. This way it's very easy to run these files:

$: node_modules/.bin/mocha -h

Thanks Joe for pointing this out!

Preparing For My Visit in Chicago

When I saw Peter's comment on my previous blog entry I realized I can't send a short response. It deserves an entire blog post, so here it goes.

Two years ago I was very unhappy with my job. I used tools and languages that did not excite me and worked on projects that I was not interested in. Reading Chad Fowler's book, The Passionate Programmer book did not help me much either. I reevaluated my life and I realized I can't spend 8 hours a day doing something I am not passionate about.

I was willing to go part time, work only 3 days a week and use 2 days to visit other companies. My wife supported me as she saw how unhappy I was when I got home from work every day. It never happened: I was able to score a Ruby job and I did not have to go to extreme measures to find happiness.

Later on I planned taking 4 months off and not work at all. I wanted to dedicate my time off to learning, visiting companies in western Europe, working on open source software and spending some time with my family back in Europe. Since I am the only person in my family who gets a paycheck, the 4 months off did not fly so well with my significant other.

I planned on visiting Hashrocket for a few days in Jacksonville, FL early March, but unfortunately that did not happen.

After playing so much with the idea I felt I was ready. I was willing to take unpaid leave for 4 days just to visit companies this fall.

Two people helped me to get in touch with the companies I visited there: Corey Haines and Michael "Doc" Norton. I worked together with Corey at a large insurance corporation and I think we met sometime in 2006. Doc led the studio side of LeanDog up until recently. So yes, I did know both of them. But not knowing them would not have stopped me, I was ready to reach out to the companies as well but I figured doing someone the intro for me would help me.

Not knowing Corey should not stop you, go ahead and ping the companies you're interested in visiting. If they reject your visiting idea I am sure the place is not worth checking out.

I have attended a couple of Coderetreats already. It's a fantastic way to get to know other developers and learn from it. The experience of visiting companies is different. The developers were up against real tasks, against real dedlines and could not afford throwing their code away after every pomodoro session. Both Coderetreats and visiting companies are great, the experience you get out of the latter is different and I think that's the key here: you learn something else.

A (Mini) Programming Tour

The TL;DR version

If you can't go to conferences, try to visit companies. Even in your own home town. You get to know many people and learn a lot from fellow developers.

---

I spent the last week in Chicago visiting four different companies, shadowing, talking and pairing with fellow developers. A good friend of mine shared his condo in downtown for a couple of days which made this trip very affordable for me. Here are the companies I visited and a brief summary of what I saw there:

:: TrunkClub

I did not know much about Trunk Club up until a couple of months ago. I reached out to Corey Haines seeking companies I could visit and he suggested them. They have a small but very talented group of developers building their - mostly internal - apps in Ruby on Rails. I shadowed Corey Ehmke on the project he was currently working on: he tried to come up with a recommendation engine using MongoDB and Ruby. Having seen him exercising the different algorithms I was gently reminded that I should probably brush up on my statistical skills.
Have you seen a company that has its own beer tap and wine cellar? Well, Trunk Club is one of them! At the end of the day we enjoyed the different variety of beers right in their offices. Now how cool is that?!

:: Hashrocket

I spent the next day at the Hashrocket Chicago office. They have a cute little space converted from a condo with a couple of bedrooms attached so people visiting from the Jacksonville, FL office can stay there. A large video screen is linked up with their home office where people stop by and say hello to the folks in Chicago.
I shadowed Matt Polito first who remote paired with another rocketeer. They used tmux for sharing their terminal sessions. I heard about tmux before, but I have not tried it yet. I noticed that developers even used it locally when they were not pairing with anybody else for the benefit of being able to suspend and resume sessions.
Interestingly the guys at Hashrocket are using a strategy pattern based solution to solve complex problems which is very similar to what I described in my Refactoring Workflows to Chain of Actions blog post.
They also used Google Plus for video conferencing with multiple people. I am not a big fan of social media but I'll definitely check out Google Plus for this.

:: 8th Light

The company's new office is very close to Union Station, which makes it easy for the commuter employees to get there. Had I not checked their current address on their web site Google Maps would have sent me to their former office.
I spent the morning shadowing Colin Jones, who worked on a file uploader web app in Clojure. I noticed how much more readable is speclj compared to Clojure test, I am going to switch to that! He also wrote a multi-method implementation that I only read about before.
The web app used joodo as the underlying web framework which seems very clean to me, but the views were built using hiccup which can be a bit too cryptic for a developer who spent a long time in HTML land.
I wrapped up my day pairing with an other engineer on some Backbone.js code test-driving it with Jasmine.

:: Groupon

I only visited Groupon, I did not sit down and paired with anybody there. Our host, Michael "Doc" Norton showed us around. Their office seems like a fun place and I have never seen so many 27" Cinema Displays in one room. Developers are working in small groups and everybody can pretty much find the project they want to work on.

What's my takeaway from all this?
I met with many talented developers. I learned how they work, what tool they use, how they develop software. I will give joodo and Clojure a try and will build a web app using them just to learn the language and the paradigm.
I know people in the US don't have a lot of vacation. But if you can do it, maybe just one day a year, visit other companies. The benefits are enormous!

I'd like to thank my current employer, Dimple Dough, sponsoring and helping me with this trip!