In this post I'm going to dive straight in and show you how you could use SpecFlow to automate a common Endur/Findur scenario. SpecFlow is a tool/technique that cuts across disciplines - it's used by BA's, Testers and Developers ... even project managers get something from it.
So I won't go as far as showing you the underlying code - I'll loose two-thirds of my audience - but there should be enough for developers to see how it could be done.
I'm not going to start by saying what SpecFlow is; I'm just going to go straight to an example. I'll gloss over some details on the way, but then at the end I'll tell you where I've left something unsaid.
The Requirements - Yet Another Broker Ops Service
Brokerage is where we (the trader) pay a third party (the broker) a fee for arranging a deal. It could be a flat fee (e.g $100/trade) or, more commonly, it is based the deal quantity (e.g. $100/lot or £0.05/MWh)
Almost every Endur/Findur client realises at some point that the out-of-the-box brokerage solution doesn't cut it, and they come to realise they need to either write a replacement or augment the existing one. They write an ops service (something that is triggered when a trade is created or modified) that calculates the brokerage instead of the default behaviour.
This client has the requirement to support tiered brokerage. The broker has different tiers based on the size of the trade. A bit like a bank paying you more interest if your bank balance is above £1000 (and more again if it's above £5000)... the broker's rates decrease as the trade quantity increases.
The Answer
Let's go straight to a complete test (one of many required) and then we'll unpick it. There is a lot of information packed into this screenshot, so zoom in if necessary.
You can view this as a specification of how the system should behave, or you can view it as a test checking that the system behaves as it should. I'm going to treat as a test for now... although thinking of your tests as a specification is a powerful idea (maybe one for another blog...)
SpecFlow calls this a test a scenario. Scenarios have names and they are of no functional significance; it just makes sense to give them meaningful names. The scenario names will appear in the execution report, which we'll look at later.
Given : the pre-condition
The first part of the scenario is a Given statement. It states what we expect to be true before we begin.
This client stores their broker rules in a user table and the ops service searches for the matching rule, calculates the fee and then updates the trade - with the calculated fee and the name of the rule that it uses. In reality the table is more complicated than I've shown here, but that's not important.
In this example we have three tiers. Trades with a volume of up to 840 MWh pay 3 cents per MWh, from 840 to 1000 the rate drops to 2 cents, and above 1000 the rate drops to 1 cent.
As the expected results for this test are predicated upon these rates, it makes sense to assert that they are as expected before we start booking/checking trades. That's the job of Given.
When : The action man
The middle part of our scenario is where we actually do stuff; in this case booking a trade.
In this case I'm booking a COMM-PHYS (gas) trade using a template. There are lots of fields that I can set when I create a trade and I've omitted some from the example in an effort to make the screenshots fit on the page.
For the purposes of this brokerage example we want to check that the solution
calculates the deal quantity correctly
finds and applies the right broker rule.
I could book multiple trades at this point (perhaps one that should fall into each tier) and I could test all combinations in one go. I think that's poor style, but it is merely a question of style; nothing prevents you from performing multiple actions in the When part. For example:
Then : but did it work?
Now we need to check if we got the desired result. First a trade should now exist and then we can check if the brokerage is as expected.
So what is the expected answer? We booked a trade for 1 MWh per hour for the whole of March. There is a clock change in March (sneaky) so the total number of hours in March is 743 (not simply 31 * 24). My trade is 1 MWh/h so the total quantiy is 743 MWh.
That falls in to the 0..840MWh tier, so the broker rate should be 3 cents. The answer is therefore 22.29 Euros and we expect a tran info field on the deal to indicate that the 0..840MWh rule was used.
So why didn't I just write this..?
I could. And the distinction is nothing to do with SpecFlow per se. Again it's just a matter of style. I think if you look at the first example your thoughts immediate turn to "why is there 1 * 23? ... ah yes, the one short day in March" and therefore the test is easier to understand and maintain (if clock change rules change, for example).
In fact my first example is slightly harder to implement - SpecFlow won't evaluate the formula; we had to write some code for that, but it's not hard.
And now I test something else
There are some obvious next steps. I want to look at modifying an existing trade and I want to check other "tiers".
I'm hoping that you can read the following without any further explanation
And I want to test the boundary condition; where the total deal quantity falls exactly on the tier boundary.
And that is the beauty of SpecFlow. With a brief summary of my requirements, I hope I've got you to the point where you could at least read my tests. If the requirement on how to interpret trade quantities that fall on the tier boundary changed then I'm hoping you could amend my tests.
And then stitch it all together
Three tests won't cover everything. So we need to write some more, and the brokerage requirements go beyond tiers. But I won't go through those, I'll just jump to our nightly test suite for brokerage....
Here is the execution report (from the real tests I lifted this example from). First the sumamry
This is the point where the project managers start to love SpecFlow. If you call this a dashboard they'll be so excited ....
I can drill down to look at a particular set of tests (which SpecFlow calls a Feature)
Or ...
Now I hope you can see why it makes sense to give your scenario meaningful names. If the name is descriptive you can read the tests at a high level. If you can read the screenshot above (which covers how to set minimum and maximum fees), then I hope that, although I've not explained them, you would have a reasonable idea of what the tests are trying to cover.
It's exciting when it fails
As a tester we're not supposed to admit that we like it when we find the software doesn't work. But it is pretty cool when your beautifully designed test uncovers a bug and you get a nice report like this.
If I've designed my tests properly we should be able to see immediately in what area we have a problem. In this case it seems to be with the application of minimum and maximum broker fees. So we click on the offending feature
The scenario names give more clues - it seems to be the tests to do with applying maximum fees that are failing. So let's drill in some more
I can see the actual contents of scenario that has failed. I have set the brokerage at 2 Euros per lot with a maximum fee of 10. When I set the number of lots to 6, the fee shouldn't be 6*2=12 because that exceeds the max of 10. But that seems to be what is happening
Bug found and the resulting bug report is
a) Going to be very precise as to what has gone wrong
b)Reference the automated test - so the developer will know when his/her fix is complete (and hasn't broken anything else)
What I glossed over ...
You need to write some code
There is no free lunch; we actually have to write some code to make this work. If you want to check a trade exists you now need to write some code that will check to see if the trades exist and whether the fields have the correct value. If you want to check the deal quantity on a COMM-PHYS trade then you have to write some code to do that. There might be 30 or 40 fields on a trade you might want to check at some point; you need to write some code.
If you now want to support Power deals, you need to write some more code.
And then, of course, you have to test this code. There is not much point writing tests if the test framework itself is buggy.
... Quite a lot of code
As I discussed in part 2, you need to make a sizeable investment in test automation to get a pay-off. We wrote several hundred SpecFlow steps to test Endur before we started to be able to stitch interesting scenarios together ....and we still find we need to add new ones from time to time.
This brokerage example was chosen because it's a real example from a real client, but also because it is unusually simple. I didn't show you the steps we have for enabling and disabling ops service, for truncating user tables, running tasks, etc that are needed to have something useful.
A full set of step definitions for an ETRM system can easily be > 20,000 lines of code. That's quite an investment.
It matters what version you're on
In V8, OpenComponents wasn't sufficiently rich to implement what you need. Attempting to this in V8 would be a classic case of what I talked about in an earlier post... starting down the road of test automation and then finding out that it's just not possible
In V9 the OpenComponents API is sufficiently rich to build a useful test automation framework (although you still need to result to Toolkit on occasions) but Endur will insist upon you running your tests out of its bin directory. In practise this means you can't get any of the benefits of running SpecFlow inside Visual Studio and that effectively makes it unusable. You can fight it, but you won't be winning people over with ease of use and productivity gains...
In V10 and up, things look better. OC is richer and you can run from within Visual Studio (with some tricks, but nothing too complicated).
Designing this takes a bit of thought
In essence you are defining a language for testing Endur (or a language for specifying how Endur should behave, if you prefer to think of it like that).
I've found that hard (see "Those Three Little Words (or how to design a specification language") and it's taken us a number of iterations to get the point where we have something useful and coherent.
... But it is worth it
Regression testing is critically important for a trading system (or any system), but it's also somewhat soul-destroying. I already tested it and it works. You want me to test it again?
Well, yes. Of course they want you to test again. But I would much rather test it every night automatically, rather than crack open a test plan and read "step 1 : login to Endur".....
And then there is the potential game changer
I mentioned at the beginning that you could view SpecFlow as allowing you to write automated tests or executable specifications.
Suppose you've invested in test automation and you have a nice regression pack running every night. You probably have everything in place to switch things around and start using SpecFlow as specification tool. In fact the name should make it clear that SpecFlow was designed to be a specification tool.
Now your business analysts, product owners or domain experts can start to express what they want in SpecFlow: An executable acceptance test if you like.
Now you can try Acceptance Test Driven Development. You start with the specification and start developing until it executes successfully.
In a complex trading system, this isn't easy. Your front office user might have no interest in doing this and your business analyst might be uncomfortable if he/she can't write reams of ambiguous prose in Microsoft Word.....
... but imagine if it could work?
We've made the switch a couple of months ago with a trading system at a large energy company. They had completely bought into test automation and we run a full suite of tests every night that cover everything from trade capture, through to P&L, exposure, P&L attribution ... all tested from first principles.
So they were open to the idea of leveraging this as specification mechanism. We've started to use SpecFlow as a specification tool, with BA's and Product Owners writing SpecFlow features as a description of their new requirements and ultimately the SpecFlow runs successfully and is folded into our overnight tests.
The experiment is still on-going. It's not really a technical challenge (we've got 1000's tests running every night that says the technology works)... it's a people challenge. Are people willing to change the way they work? I'll report back when we have some findings to share.
If you've tried something similar, or are thinking of trying something similar, then I'd love to hear from you.