We run 12,579 tests before every release
Every release passes 12,579 automated tests before it leaves our hands. Here is why that number matters to you.
Flowdrome runs your real work. Your actual jobs, on your own machines, at three in the morning when nobody is watching.
So we test it. A lot.
12,579 tests, every release
Before any release leaves our hands, we run 12,579 automated tests. Every one has to pass. Not most of them. All of them.
That number goes up every release. New features bring new tests, and we keep going back over the parts that already work to test them deeper than before.
Those tests currently cover 88% of our code. We are aiming for 100%.
Why it matters
Automation fails in a quiet way. It usually does not crash. It does something worse — it succeeds, and hands you the wrong data. The run goes green, the dashboard looks fine, and a week later you find out one field was empty the whole time.
Clicking around before a release will never catch that. Only a machine checking every path, every time, will.
A test has to be able to fail
We once had eight tests covering a feature. All eight passed. Then we broke the code on purpose — and all eight passed again. They were not testing anything.
So now we break the code deliberately, to make sure the test notices. If it cannot fail, it is not a test. It is decoration, and decoration is worse than nothing, because it makes you feel safe.
What this means for you
Upgrades are boring. Your workflows keep working. And when a run says it succeeded, we have gone to real trouble to make sure it actually did.
We would rather you forget about it.
— The FD Team