harriyott.com

Wednesday, July 20, 2005

Unit testing for speed?

I was just thinking about performance testing, and wondered if writing a unit test would be sensible. If a functional requirement is, for example, "Import all of the data in the crime scene XML file into the database within 0.5 seconds", a test could be written to:
  • Get the time into startTime
  • Call the ImportCrimeSceneData method
  • Get the time into endTime
  • Assert endTime < startTime + 0.5 seconds
If this fails, then the ImportCrimeSceneData method can be tuned until it passes. Once it has passed, then any future changes to the method are checked automatically to ensure that the limit isn't exceeded.

Although the idea seems like it would work, it feels like a wrong thing to do. Has anyone else done this successfully, or can anyone give me an actual reason why this is wrong (other than "it just feels like it")?

4 Comments:

Anonymous said...

In my opinion there are too many outside factors that come into play when testing performance. A Unit Test may pass on one machine and fail on another, despite having exactly the same code.

July 20, 2005 4:35 PM  
Simon said...

Thanks, that's a good reason.

I guess that could be levelled against the functional specification too. If it doesn't say what machine configuration the half second applies to, then it can't be measured properly, and therefore can't be signed off.

If it does say what machine to run on, then I suppose it could be tested on a similar machine: in this case, the database might be on a networked server, which makes the measurement doubly unreliable.

July 20, 2005 7:25 PM  
Grig Gheorghiu said...

I think it's a good idea in some situations. Recommended reading: Continuous Performance Testing with jUnitPerf. I ported jUnitPerf to Python. See Performance Testing with pyUnitPerf.

July 26, 2005 5:21 PM  
Simon said...

Thanks Grig, that's a really interesting article, and pretty much what I had in mind. I've read a few of Mike Clark's things, and they're usually pretty good.

July 26, 2005 6:27 PM  

Post a Comment

Links to this post:

Create a Link

<< Home