Monday, June 8, 2009

Comments to Robot Framework review by Gojko Adzic

Gojko Adzic organized Agile acceptance testing tools round-up on 27 May 2009 at Skills Matter in London. One of the tools he presented was Robot Framework and he published Robot Framework review on his blog afterwards.

Gojko's review is very well written, especially thinking that it is mainly based on the documentation on the project pages. The review is also pretty positive which obviously makes me happy. I had a change to see his text already before the presentation and some of my comments are included in the blog post. After re-reading the text I decided to add few more comments here. The comments below are organized into same sections as in the original review.

Keywords

It was great that Gojko spent some time to explain the keyword-driven concept behind Robot Framework and included my comment about how it makes the library API simple. I hope he had also shown that although this approach is simple, it's also very powerful and it is possible to create different kind of test cases using it. Robot Framework Quick Start Guide has very good examples of "normal" keyword driven tests, data-driven tests, and tests using the behavior-driven development (BDD) style.

Organising and editing tests

This is a good section about creating tests. Unfortunately the example, taken from the examples we use for explaining complex features, is pretty technical. For example the earlier mentioned Quick Start Guide has examples that are much better as tests. Normally when I talk how tests are organized into files and directories, I also mention how trivial it is to put them into a version control system. This is probably obvious to everyone reading the review, though.

Filtering

This section discusses tags and how they can be used for filtering tests and marking tests critical. Gojko's examples are very good, I especially like that they are from his own domain, and they illustrate these important features very well. The only thing I wish was mentioned too is how tags can be used to get statistics about passed and failed tests.

Compared to other tools

Gojko writes that Robot Framework is best suited for scripting tasks. Although I totally agree that RF suits very well for traditional scripted test automation, I'd like to highlight that creating also other kind of tests is possible. This is illustrated by the examples in the Quick Start Guide, and the executable requirements we created when developing RFDoc tool act as more realistic examples. In my opinion it is also a huge benefit that the same tool can be used for "normal" scripted testing, for creating BDD style executable specifications, and for constructing tabular examples with only input and output values.

One of the many points where I fully agree with Gojko is that Robot Framework can be used for creating domain specific languages. This can actually be done on many levels. For example SeleniumLibrary has its own web specific language with keywords like Click Link and Title Should Be, application specific language could have keywords like Input Password or Welcome Page Should Be Open, and finally on the specification level the language often sounds something like Given a user has logged in.

Reports and logs

The only bigger subject missing from the review was a section with this title. Reports and logs are something where Robot Framework is, at least in my opinion, ahead of other acceptance testing tools by a wide margin. The high-level report can be used for visualizing the overall status, detailed logs give accurate information about the test execution, and XML outputs can be easily integrated e.g. with CI systems and even combined to generate higher level reports.