I am actively programming the PalmOS for about half a year now-and went through many beta testing cycles with mostly moderate sucess-as 1.1 versions demonstrate. However, most mistakes made on my end are avoidable-if one only knew how. This article series will cover most of the things I deem important.
Random testing is useless-structured testing rules
Given enough eyeballs, all bugs are shallow-the Linus law
This classic law of open-source programmers sounds reasonable. But: how many testers can you afford in terms of time, management effort and money? Most developers go for 10-20 people, of which about 75/100 never get active. Thus, you are left with five to ten beta testers who will probably all do more-less the same routine tasks and are unlikely to perform stress tests,… .
But-most bugs cant be discovered by just doing a quick overview. You may find quite a few-but there always is a rest risk that some obscure bug manages to slip in.
Compare this to a structured approach. A todo list covers the basic testing procedures, making sure that at least the basic functions are tested on each machine. And once beta testers know the frame of operations, they can develop their own test cases easily.
I know that test cases need to be developed, but trust me, the hour invested pays out!
Tune in soon for the next part… What do you think so far?
The art of beta testing-Part 1
Hi Tam,
ok, beta and frontend-testing is one form off testing. Which testing methods do you use in your development process? Do you implement unittests or some other kind of component test methods?
Hi,
I started the external tests earkly in the alpha process. That and an occasional code sanity check were all my measures up to now.
Now, I use a strictly planned betatesting that happens in-house on various machines and code sanity reviews on paper.
How do you do it? Any info on other methods?
Best regards and thanks for talking back
Tam Hanna
[...] Welcome back to our art of betatesting column. Just in case you forgot what the first part was all about, feel free to read it up here: The art of betatesting-structured beta testing Completely outsourcing tests is dangerous Man, your old-style. Having five handhelds at home is stupid! I dont have a single one at home or in my office-Developer of Censored, on ICQ This impressive person apperently trusts his betatesters even though he does not give them test cases or anything. If he has extremely committed testers, this will work fine. However, if not, goodnight. A beta tester usually is not paid (much) and is a plain user interested in either the new program, getting a serial for free or showing off to his friends that he runs the latest and greatest apps! His interest in your well-beeing is rather low. Exceptions exist-guard them like a treasure, people! Anyways, a beta tester has a higher probability to overlook things than the original developer has. It doesn’t have to be lack of interest or typing effort, he can also treat this as o.k. as he discovered a workaround. Anyways, there is an old german proverb saying that control is better than trust. Keeping at least a mid-end box like a Tungsten T or E2 in your labs definitely wont bust your budget-but it would keep you at least theoretically capable to test your software yourself. If you are too lazy to do it though,… Got something to add? [...]
You should be careful to distinguish two meanings of ‘random testing’. The first meaning — unstructured, unplanned, arbitrary testing — seems to be the one you mean. But there is another meaning: automated testing using very large numbers of test cases generated by a randomized test generator. This latter approach is most definitely NOT useless — it will find bugs that structured testing with manually constructed test cases will have a very hard time finding. Indeed, experience is that any non-trivial program subjected to such testing for the first time will quickly break. This includes otherwise mature programs that have extensive test suites, such as gcc or Mozilla. Microsoft estimates that 15-20% of the bugs found during development of their products come from high volume randomized testing of this kind.
Hi Paul,
thank you very much for the comment!
Actually, sth like that exists for the PalmOS too, sorta. It is called Gremlins, and is a UI-Stresstester…
Best regards
Tam Hanna