firebrigadeWhat we Report
Firebrigade uses Tinderbox to run tests and submit the results back to Firebrigade. (If you'd like, you can run your own Tinderbox.) Each Tinderbox reports on the test status of the newest gems it can find in the RubyForge gem repository. Tinderbox ignores gems it can't install, such as gems that are compiled for only one platform.
After installing a gem Tinderbox tries three different ways of running a gem's tests:
- If a Rakefile is present, run rake test
- If a Rakefile is present and rake -T contains "rake spec", run rake spec
- If a Makefile is present, run make test
- If a test/ directory is present, run ruby -Ilib -S testrb test
- If a spec/ directory is present, run spec spec/*
Tinderbox makes a best effort at testing gems and won't stop until it comes across something that looks like test output. For example, if a project has a Rakefile that is missing a dependency like rcov, but the tests can also be run via testrb, Tinderbox will continue past the Rakefile error to run testrb.
Tinderbox only waits so long for a gem to install and run its tests. (All dependent gems are downloaded and cached in advance, so download time is not counted against the gem.) If a gem takes too long to run (120 seconds by default), the gem is marked as failed.
After running the tests, Tinderbox then looks in the test results for a test/unit summary like this:
137 tests, 551 assertions, 0 failures, 0 errors
or an RSpec summary like this:
5 specifications, 0 failures
Either of the above summaries would indicate a passing gem. If the gem has either failures or errors the gem will be marked as failed. If the gem didn't run any tests, ("0 tests, 0 assertions, 0 failures, 0 errors" or "0 specifications, 0 failures") it is marked as failed.
Tinderbox also reports gems that appear to depend on Rake (have a Rakefile) or RSpec (have a spec/ directory) but don't list Rake or RSpec as a dependecy. Tinderbox is nice and goes ahead and installs these gems for the gem author if they've neglected to list it.