When Symfony2 was created I first learned about the functional test, which is an interesting type of test where everything about your application is as real as possible. Just like with an integration or end-to-end test. One big difference: the test runner exercises the application’s front controller programmatically, instead of through a web server. This means that the input for the test is a Request object, not an actual HTTP message.
Drawbacks of functional tests
This approach has several drawbacks, like:
- Faking a
Requestobject means you don’t know if the code also works in a production environment, when an actual HTTP request will be provided as input. - Inspecting a
Responseobject instead of a real HTTP response message comes with the same risk, but my guess is that it’s a less severe risk. - Since you have access to the application’s service container you may be tempted to open the black box and replace all kinds of services, or otherwise influence them. I recommend doing no such thing.
- You are likely to end up testing all kinds of domain-related behaviors through thick layers of unrelated code.
This obscures the view on the real issues and makes it hard to figure out what’s wrong, if anything pops up. Combining this with drawback 3 you often find yourself rummaging around in all the stuff that’s on the inside of your application, lengthening the feedback loop between building something and finding out if it works. Because these tests end up being white-box tests after all, you also end up with many cracks; it will be easy for anything to fall through them and you’ll only find out if that happened when the code is already running in production.
Another issue I see with functional tests is that the assertions that are made are often something like: “I see in database”. This too is like opening the box and looking inside. I’d like to follow this reasoning instead: if, as a user of the system, I expect a request to produce some kind of effect, then I should be able to notice this effect as a user. And as a user I can’t look directly inside the database. There must be some other aspect of the system that was changed when I made that request. E.g. after registration I can now log in, or the product I added to my basket actually shows up on the “my shopping basket” page.
In some rare cases the desired effect can’t be observed by a user. For instance, when you expect a certain message to be published to a queue. Well, maybe a functional test isn’t the right place for such a check after all, and you can prove that the message will be published using a combination of other, smaller tests. But if you still want to do it, you’d have to do a quick peek into the box after all. Sometimes that’s just how it is. But always look for ways to prevent this, and let your application remain the black box that you poke at with HTTP requests only, making only assertions about the response you receive.
Having discussed many of the downsides now, let’s not forget the good parts: we don’t have to deal with a web server or run all those slow browser tests without getting a “90% okay” approval from our tests. We can accept the “risk” and work with functional tests instead of true end-to-end tests which force the test to send an actual HTTP request to an actual web server and wait for an actual HTTP response. We can benefit from some options for peeking-in-the-box. If we can keep the number of times we do this to the absolute minimum, we will end up with high-quality tests that don’t fail for just any random reason, like concurrency issues.
Bringing the database in the right state
One thing that turns out to be quite hard when writing functional tests is getting the database in the correct state for running the tests. In fact, we have to get it in the right state before each test, so that tests can’t influence each other when they use the same database tables. Ideally I’d always use the actual steps that a user would take to get that data into the database, instead of loading data directly into the database using some kind of fixture generation tool. But for some data, this is just impossible. That data needs to be there from the start. Sometimes it’s because how the application has been designed; or maybe nobody realized there was a problem until they started writing tests (we call it a “legacy application” then ;)). Sometimes it’s a hint that this data-that-is-always-there should not be in the database but in the code instead, since the code is by definition always-there. See also About fixtures.
Before running a functional test you have to get the database into the correct s
Truncated by Planet PHP, read more at the original (another 6953 bytes)