• 04
  • Mar

Introducing Chefup

A few weeks ago Github released Boxen, a “tool for automating and managing Macs at GitHub”. At Mynewsdesk, with more than ten developers and a handful of projects, we can see the need for this kind of software. When summed up, many days have been spent trying to get developer laptops back in a working state for one reason or the other. We also had a lot of smart configurations, custom hacks, different versions of databases, sometimes shared with some collegues but without a central repository of collaboration for this kind of configurations.

At Mynewsdesk we’re using Chef for setting up and configuring our server environment. Boxen didn’t feel like a perfect fit for us, since it’s based on Puppet. Puppet and Chef are the two main competitors in this area of operating system configuration management.

To solve this we’re now presenting Chefup. The pipe dream of Chefup (and Boxen) is to be able to install a full developer machine with a single command. But more than that it also provides a way of keeping the machine in a working state, with all key components fully documented. As a bonus this can all be done without running sudo and wreak havoc with your system. Some cookbooks may themselves run sudo though, such as homebrew and pow.

Initial Setup

Firstly, this is still beta software! Please make a backup of your machine before running it and be prepared for massive losses of data, corrupted files and possibly the plague. Report any bugs to The Chefup Issue Tracker.

  1. To begin with, clone our open source project from Github; 
    git clone git://github.com/mynewsdesk/chefup.git

    If you still don’t have git installed you may want to curl it instead;

    curl -L https://github.com/mynewsdesk/chefup/tarball|tar xz

    That will get you a version without the .git directory but you can make a proper clone later on, when you have access to git.

  2. To be able to compile you also need to install XCode from App Store.You’ll need to agree to the license by running; 
    xcodebuild -license
  3. Now create a Cheffile. That’s a list of external Chef cookbooks to be installed. 
    cp Cheffile.example Cheffile

    If you work at Mynewsdesk and have access to our private repos you may uncomment the chef-mynewsdesk repo to setup our apps.

  4. After this you want to decide exactly what Chefup will install for you. This is done in the node.jsonfile. An example is provided for you so you want to begin with copying it. 
    cp node.json.example node.json

    Now have a look at that file. In the header you enter your personal data in form of name and email. Then you choose where you want to keep your code, where the default is ~/dev.

    After that have a look at the run list. That’s a list of packages, or cookbooks in Chef terminology.

  5. Now, logged in as your normal user, run; 
    bin/chefup

    This will install the bundler and librarian-chef gems, the homebrew and dmg cookbooks and any cookbooks listed in the run list in node.json. If you have chefup-bin in your run list you will have chefup in your $PATH after the first run, so it will be enough to just run chefup.

    Any backups of overwritten files are kept in tmp/backup. Any installed apps (with an .app suffix) will be installed in ~/Applications to avoid needing root permissions.

Living with Chefup

After the installation it is safe and even encouraged to run chefup as often as you want. It will then overwrite any changes you made to the Chefup controlled files. Therefore, if you install for example the bash cookbook you will not want to edit ~/.bashrc manually any more. Custom changes will instead be put in ~/.rc.d/. More information may be found in the README.md of the respective cookbooks under site-cookbooks, such as chefup/site-cookbooks/bash/README.md.

If you miss something, feel free to add a cookbook. Either as a new git repo and adding it to Cheffile. It will then be installed to cookbooks in the next run. Or by cloning the chefup-repo, adding cookbooks to site-cookbooks. Then please feel free to send a pull request to have it added to the official repo, if it could be of use to others.

Posted by Leave a comment Posted in git, Ops
  • 13
  • Sep

The pair programmers prompt

At Mynewsdesk we love pair programming and do it all the time. To avoid manually changing our git configurations we use tools like hitch and git-pair. But I have found that we are forgetful and sometimes leave our pair configuration untouched when we stop pairing. To stop us from committing bugs in other people’s names we thought it was a good idea to display the pair info in the prompt along with the current git branch.


Just add this snippet to your .bash_profile.

Posted by 1 Comment Posted in git
  • 29
  • Jun

Adding a Scout plugin in less than 30 minutes

On mynewsdesk we send a lot of emails as discussed in a previous post. We have a queue for outgoing email deliveries. Depending on our customer’s plan, the emails are sent with different priority using delayed_job.

Customer Support asks us about the status in the different queues “When will customer X’s email be sent?”. We resolve this question by doing manually SQL-queries against the delayed job table. A boring, tedious task… Scout to the rescue!

With Scout we can easily provide Customer Support with this info right away by creating a plugin. And it’s super easy!

I looked into the offical delayed_job-plugin, tweaked it a bit, uploaded it to our utility-server (which runs delayed_job). Scout automatically found it, and we could start using the plugin right away! See instructions for uploading your plugin here.

Number of pending mail jobs for different customer types

 

The source code for the plugin can be found here.

Posted by 2 Comments Posted in Ruby on Rails Tagged , ,
  • 08
  • Jun

Migrating to Sendgrid

Mynewsdesk sends a lot of email. Transactional emails (signup emails, activation of account, payments etc), emails from our users to their own network and alerts to journalists, when a company publish a pressrelease.

Issues with current setup

One common question from customer service is about status for an email delivery. Until now we have solved this by manually looking in the mail logs of sendmail, which could be a bit tedious and something that customer service can’t do by themselves. We didn’t have any nice statistics overview as well with open-rate, click-rate and so on.

Moving to Sendgrid

Sendgrid is a Software as a Service (SaaS) platform that provides a complete solution for sending emails and getting real-time statistics about the deliveries. It also provides an API where you for example can use their Event API to call your application for status about each mail you send.

With Sendgrid any person can log in and see general stats, bounce reports and status about a certain delivery. You can’t see the contents in the mails though, you have to use their BCC-app if you also wants to see the contents for each mail.

Stats for Forgot Password emails

Bounce stats

Searchable email activity

 

Integration

When migrating to Sendgrid we didn’t move all emails at once. We started by categorizing all our outgoing mails and move a category at a time to sendgrid. Using sendgrid is very easy, you just update your smtp-settings to point to your sendgrid account and voilà – integration is completed. You can also tag your emails with sendgrid categories to be able to get statistics for a group of emails.

Here is our integration code with sendgrid. (Thanks Joakim Kolsjö and Henrik Nyh with idea of how to find calling mailer action!)

Posted by 7 Comments Posted in Uncategorized
  • 31
  • May

Steve Jobs says boom

We have two lab days per month here at Mynewsdesk and one thing that never seems to stop receiving love is our chatbot Flower (for the Flowdock service). To keep the tradition alive I have now added a new command to the bot: The Steve Jobs boom command!

Type !boom in the chat and the bot will play a random clip of when Steve Jobs says boom through the office speakers.

 

Posted by 1 Comment Posted in Services, Workflow Tagged , ,
  • 29
  • May

Vim Concepts

Introduction

At Mynewsdesk we develop Ruby on Rails applications and the developers may use any text editor they like. This has led to a myriad of different editors in use, from Textmate and Sublime Text to Emacs and Vim. Personally I prefer the Vim editor and have been a user since 15+ years.

There are many ways to approach and learn Vim but surprisingly few tutorials have a closer look at the main concepts of Vim. I think understanding these takes away some of the confusion when using the editor.

Modes

Vim has six basic modes; normal, visual, select, insert, command-line and ex-mode. These are basically different states of the editor and a basic cause of confusion for many users.

When starting up the editor it’s in normal mode, which means that most unmodified keys execute a command, rather than inputting text as in most other editors. This makes sense, since most of the time in an editor you don’t really enter text, rather you navigating, moving, changing and searching text.

To enter text you have to enter the insert mode, to select text you enter visual mode etc. There are also six more additional modes which are variations of the basic modes. Read more about the modes by typing :help vim-modes.

Registers

There are a bunch of registers, which could be compared to temporary variables in a programming language. All the registers has a one character long name and to use them you prefix them with a double quotation mark, such as “- or “x.

Some of these registers are updated by Vim itself, such as “/ that contain the last search and other are user updatable, such as “a to “z. You may also use these by prefixing a command in normal mode with a register, such as “/p for outputting last search or “ayy for copying current line into the a-register.

Read more about the registers by typing :help registers.

Buffers and Windows

A buffer is a file loaded into the memory of the editor. This would not be of big use without a way to actually view and edit it, and that is done in a window. There is not a 1:1 relationship between them though, a buffer may have one or more windows attached to it (called an active buffer) or no window at all (a hidden buffer), and a window may be with or without a buffer.

When started Vim shows one window but they may be split vertically or horizontally into more windows. You may then move around between them or move around the windows, resize them, open different buffers in them etc. There are ways to run commands in all open buffers, to hide them, remove them from the buffer list, unload them from memory and more.

The buffers may be listed with the command :buffers and you may read more about them and windows by typing :help buffers.

Tabpages

A quite recent addition to Vim is tabpages, added in Vim 7. A tabpage is a collection of one or more windows. This is a nice way to open a buffer, without changing the current window setup. It may for example be used when temporarily opening up another project with another set of files.

It may cause some confusion that the window splits lives in the the tab, rather than each split having its own set of tabs as in some other editors, but used as mentioned above it’s one of the strengths of Vim tabpages.

Read more about the modes by typing :help tabpage.

Other concepts

This has touched some of the basic concepts that are most commonly used. There are also a bunch of other really useful ones, such as folds, quickfixes, undo branches, text objects and tags. These and much more is described in the Vim help, please go ahead and navigate it with the :help command.

Posted by Leave a comment Posted in Editor Tagged ,
  • 23
  • May

Writing commit messages

Writing correct, nice and to-the-point commit messages is not always easy. Sometimes when we have a lot of changes to commit, it’s easy to write some nonsence just to have it commited and pushed before clocking out at the end of the day. Or sometimes we’re just being a bit lazy or unimaginative.

Like these ones.

  • fix for site fuckup (Jun 10, 2011)
  • added old css (Nov 9, 2011)
  • Fixed weird issue with subdomains (Oct 20, 2011)

Is it easy to understand what the commits are doing?

A good commit message can sometimes save us the time of having to read the code or do a git bisect to understand where a bug might have originated. If done properly over a period of time, the red thread of the features also becomes apparent.

And then there are some that are just weird.

  • Trying to change so that on the publish/edit page the categories are displayed in the locale of the user with the pressroom site (selected_site) locale used as a fallback. This means that in the finnish pressroom a swedish speaking user gets the category names in swedish and only in finnish if there is no swedish translation. Removed a number of with_locale(selected_site) blocks. I also had to change the JavaScript locale in publish from selected_site to I18n.locale (current locale). The Category#on_site scope now returns translations both for the site and the current locale. The new Category#name_with_fallback(locale) method is used when displaying the category names. It goes to the current locale first and then then falls back on the locale argument (the site locale in publish). TODO: it’s confusing that we now have Category name, name_with/without_site and name_with_fallback methods. Time for a cleanup. (Jun 28, 2010)
  • ooooo, this HURT! (Feb 7, 2011)

I guess there are as many rules and opinions about this as there are developers, but this is something I personally think about when I write my commit message.

  • Always write about what the code changes, not what you have done (e.g. never start with “Added …”)
  • Always write in present
  • Capitalize the first row, without periods, and try not to exceed 50 characters (I’m not a nazi on this)
  • Don’t be hesitant to add an explanation as to why the changes have been made (on the third row, leaving the second blank). In fact, if you’re thinking about it, that’s probably a good sign that you need to do it.

If you think I have missed something vital, please comment with your own tips and tricks to write the ultimate commit message!

Here are some of my personal Mynewsdesk Hall of Fame commits:

  • don’t negate, that is a negation (Nov 2, 2011)
  • undefined fucking method (Jan 19, 2012)
  • Pizza! (Dec 15, 2011)

Update: As requested by @patrikstenmark, here’s a few examples of good commit messages:

  • Acceptance test for access signup process (May 21, 2012)
    In this commit, we added a capybara acceptance spec file with two tests. There wasn’t any need to write that we added it, there hardly ever is.
  • Kundo iframe needs transparency attribute in IE (March 29, 2012)
    We use iframe based Kundo for support to our customers on top of a div with a background image. We added an attribute to the iframe to allow it to be transparent in IE. Being such a small change, we simply wrote why the change was made.
  • Refactor weekly recipients to dashboard email to own class (May 7, 2012)
    A refactoring commit that broke out some logic for our weekly summary email into its own module. Small commit message that is to-the-point and no messing about.

Why I think these are good examples is that they adhere to their context. Bigger changes often need a more general summary, smaller changes or bugfixes might want to focus on the why. I don’t think there’s a silver bullet to writing them, just some guide lines and a lot of different contexts you should be aware of and adjust to. Anyway, that’s only what I think.

Resource links (thanks @henrik and @kalasjocke for pointing out the lack of it):

A note about Git commit messages by tpope.

Commit message guidelines for Git itself, follow them or Linus will come and kill you with a hammer.

 

Posted by 4 Comments Posted in git
  • 21
  • May

Stockholm Startup Hack — Trackr!

Last weekend me, Jonas and Martin spent 12 hours hacking at the Startup Hack in Stockholm. It was a great hackathon, and we actually managed to get some kind of a product out!

At breakfast we discussed our initial idea to create some kind of viral, mobile friendly, Facebook enabled, lame game. But the idea for the game was too bad to present to the sophisticated crowd (and also, we can’t really program games), so we had to come up with something new, really fast.

Trackr!

The new plan was to create something useful we could use in the office — realtime website stats! We currently use the great service Chartbeat to track number of visitors currently online, top referrers and other useful stats. But once you want to track more than 1,000 concurrent users the service becomes quite expensive.

We created something similar; using a tracker script inserted into the customer web page, we poll active users every 5 second by generating an image element pointing to our app. This will require a fast, asynchronous backend to handle the traffic from multiple sites!

The Setup

Hosting on Heroku can get you quite far for free, perfect for our lab! The services we choose to get up and running was:

  • Tracker app with Thin webserver running async_sinatra to track visitors asynchronous. Exposes a JSON API.
  • Redis, perfect for storing and generating quick reports of “online visitors”. Redis To Go is free for 5 MB memory!
  • Simple frontend/reports app running Rails to “signup customers” and view your report.
  • Unicorn web server for the reports to run multiple processes in the single, free Heroku dyno. (The realtime graph needs to update every second or so.)
  • Great chart lib Highcharts to visualize our live stats!

The result

Our simple frontend app should actually work if you want to try it out! Its currently hosted at http://trackrapp.herokuapp.com/.

We tried this on Mynewsdesk, and the results are quite good! The low capacity of a free Heroku app, and 5 MB limit in Redis makes the reports not perfect (actually, only about half the traffic is being reported…). But in the current state I think the app could accurately track a website with fewer online visitors than Mynewsdesk (1000+ in peak hours).

Feel free to try it out!

Posted by Leave a comment Posted in Uncategorized
  • 11
  • May

Scout Rails app monitoring

I saw this tweet from Scout just before leaving the office yesterday:

It looked really promising, and we really like scout but are still using New Relic for the monitoring of our Rails app. So I sent them an email and the first thing we did this morning was to deploy the new Scout Rails monitoring of our Rails app.

We’re happy with it so far, it can surely replace New Relic:

Scout Rails app monitoring overview

 

Posted by Leave a comment Posted in Analytics, Ruby on Rails, Services Tagged , , ,
  • 17
  • Oct

Testbot – Distributed testing

TL;DR

If you have a lots of tests in a rails app that is annoyingly slow (+10 minutes), try testbot – a tool for distributed testing

Testbot

What?

Testbot is a tool for running tests distributed over several machines. Mynewsdesk test suite, ~1900 examples and growing strong, takes about 20 minutes when running on my local machine (a weak macbook air). By using testbot the suite takes about 3 minutes. Thats almost an performance win by 700%! Costs money though (ducks!)

How?

Testbot is composed by three parts.

Requester

The requester is the local machine that initializes the test-suite to be run with testbot. The requester will send its environment (rsync) to the master and waits for the test suite to be run.

Master

The master will split the test-suite into smaller parts and let runners set up the environment (rsync, bundle install) and run the subset of tests. The results from the runner will be sent back to the requester.

Runner

The runner is our work horse that will run a group of tests and sends the result (stdout) back to the master.

For more a detailed explanation see testbot’s documentation.

Our setup

We are using AWS EC2 to run our master(1) and runners(11). All our instances are of the type medium (2 cores, 1.7 GB memory)

Our requester

Every developer’s own computer

Our master

Some standard linux image offered by ec2. The testbot-master requires ruby and the testbot gem installed. For us, this machine also has a cron script that schedules when runners are to be run (office hours)

Our runner

This machine has all the required software to run our rails application in a test environment. Our rails application depends on solr and postgres, so this software is also installed.

How fast can the test suite run?

Distributed testing allows you to increase the number of runners to decrease the time it takes to run the whole test suite.

Bottlenecks for testbot is the time it takes to rsync the environment from the requester to the master, then rsync the environment from the master to the runner and lastly the time it takes for the runner to start the rails environment and run X number of tests.

For us this means that it should be theoretically possible to run our whole test-suite in seconds(!).

Our gotchas

Environment setup for the runners

In the beginning the runners didn’t run in a stable environment. For example solr didn’t always startup properly and we had to use monit to ensure solr was running.

Cache store

During our upgrade to rails 3 we got errors when reading from Rails.cache

Errno::ENOENT:
No such file or directory – /tmp/testbot/mynewsdesk/tmp/
cache/Site.local_sites20111013-3227-17jr2c4-0.lock

The problem could be avoided by using memory_store instead of file_store.

“It only fails on testbot!11! FUUU”

We have had quite a lot of tests that only have failed on testbot. Almost all these errors can be traced back to the fact that the test doesn’t start in the state that we expect. Some test changes a global variable/database-table but doesn’t resets the environment when the test is finished (side-effect™). Similar to this is that a failing test doesn’t ensure a clean environment when test starts.
Failures due to incorrect data in solr is one of our classics.

Tips

Simplify the process of updating runners

Testbot will handle your gem changes automatically if your are using bundler. Prepare yourself to make changes outside the rails stack smoothly. EC2 AMI’s functionality will solve this problem for you. Chef or Puppetmaster is an even better strategy if you someday want to migrate to another plattform.

Split big tests into smaller

Large test files will limit how fast your test-suite can run with testbot. If a single test file takes two minutes, your test-suite can never be faster than two minutes regardless of how many runners you add.

Final words

A lot of rails project would definitely benefit from testbot. Is there any other distributed test frameworks that are awesome?

Posted by 5 Comments Posted in Ruby on Rails, Testing Tagged , ,