So I’ve been looking around at open source project hosting websites, and there’s one thing I think is really, desperately missing from nearly all of them (github, bitbucket, and google code. All look equally good to me.)
Test framework integration.
Now, I realize they can’t offer to run the tests for me (too limited/abusable), but why not just write down a format, and let me upload test results with an API? That’d still provide:
- A nice web interface to test result information. (“4 failures” *click* “oh, those.”)
- History of failures / fixes; differences between branches.
- One click link to the test code that failed.
- Maybe some integration with issue tracking (by noting in the test metadata what bugs they’re related to)
- Even better: provide not just for true/false success/failure, but also numerical data, like performance, so benchmarks can be tracked over time.
- All kinds of other wonderful things I’m too lame to think of.
Of course, there’s another problem: I can’t even find any stand-alone tools that let me do this. There’s all kinds of tools out there for bug/task management; where are the test result database tools?
Isn’t this a dead-obvious thing every project should have these days, like version control and issue tracking? What am I missing?
(I know TAP exists, but it doesn’t really provide any of the necessary metadata in its protocol, and none of the TAP consumers seem to provide any of the above features, that I saw.)
Update: It appears the keywords I was missing were “continuous integration.” I’ve now found TeamCity, Husdon/Jenkins, etc. So shift the question: which of these do you prefer, and which one accepts some reasonable format of test output that I can give it? My tests are in our own test framework by necessity since they’re all in our own language…
Although, the above still stands: those project hosting websites should offer an API that a continuous integration server can talk to.