We are in need of supertesters to promote packages on Extras. http://wiki.maemo.org/Testing_Squad

This is a crucial function which is currently suffering because the list of supertesters and promotion rules have not been updated in a long time. http://lists.maemo.org/pipermail/mae...ry/028825.html


https://garage.maemo.org/projects/qatesters/

Of the current list, I propose to keep Andre Klapper and Harald Schmitt because they continue to be involved. If anyone has been involved and I missed it, please speak up. I'm sorry and you will be kept as well.

For new supertesters, we should have people who have a demonstrated interest in maemo and ability to test maemo packages. It is vital that supertesters are trusted members of the community. A single supertester will be able to promote someone else's (not their own) package. Please feel free to suggest different or more objective criteria for supertesters.

So far the following have been nominated:

Pali
freemangordon
nicolai
merlin19xx
MAG

From my side, I completely trust the nominees.

Thank you for pushing this initiative.

The problems I see with the nominees is that they are:

1. developers, so instead of coding they have to do QA
2. part of CSSU team and KP (AIUI the most important comminity projects by now), again, instead of coding/maintaining CSSU and KP they have to do QA.

It will be much better if we have other QA people nominated too. And there are good candidates for QA team, look at the adeclock thread or replacement media player thread (for example).

Agreed!
Just what I was thinking when I saw the nomitated users

Wee question:

Does a criteria / documented process exist for the testing?

I'd hate to think it was a free-for-all chuck-it-on ala batterypatch, or conversely a box-ticking exercise ala Ovi.

I think these testers shouldn't be developers but normal users which have knowledge of the workings of maemo/linux in general and are able to give good and comprehensive feedback. I am looking at Estel or Mentalist Traceur. Ofcourse only if they want to

For the rest of my opinion I refer back to Freemangordon's post (#3)

I can nominee myself, although I have to warn that I don't have enough time to test everything that pops out in the repositories.

Now I try to test as much as possible and report from time to time (you can check my votes) but, as said I can't promise to dedicate too much time to the task.

I think developers as supertesters is OK, as they will be sensitive to what to look for.

Also, I think volunteers should be asked for first, then voted on, to save any embarassment to someone who may be "drafted" but cannot for time, burnout, or some other reason become a supertester.

gregoranderson: there is a Quality Control procedure somewhere in the wikiswamp, but some of its criteria are assessments of issues not immediately understood or handily measured by casual users - optification, battery consumption, etc.

Is it too much hassle to have a community vote?
For example, nominations open until 23:59 xTZ* on 7th Mar 2012, voting 00:00 xTZ on 8th March until 23:59 xTZ on 14th March 2012?

The notion of super-testers is a very good one. However, I have to I agree with freemangordon's comment re: administrating testing vs responding to tests (i.e. fixing bugs & writing/editing code). (post #3)

While super-testers should be code-aware, perhaps they should not be responsible for contributing code to the piece of software they are testing (and trying to promote through the QA system). The reasons for this are
(a) time is finite so it's very much a case of, either administer the bugs or write code; a programmer will, quite rightly, believe he or she is of more value to the cause by writing code - especially if the bugs pertain to software he/she has very little to do with.
(b) managing bugs is not about programming itself, it's about describing problems and finding out whether/how they can be resolved.

Another reason for Not having key developers administering bugs is that a significant bug for many end-users is the absence of documentation. Preparing documentation is time-consuming so arguably someone whose strengths are writing C/C++ etc. are probably best directed to creating software, not paperwork! On the other hand, decent explanation of how to use software is truly vital. Indeed, I would consider it part of the UI, especially for terminal programmes where there is no graphical UI; "<command> --help" isn't always that enlightening!

Once a tester has investigated a programme, he or she should be able to verbalise how to get the most out it, as well as converting (sometimes terse) developer comments into cohernt prose. The most technologically averse user can be surprisingly helpful in doing this.

There is obviously a problem when developers no longer attend (for want of a better word) the community. In this case, talented coders may be needed to coerse the problematic parts of programmes that are languishing in -devel or -testing to get them functional enough to do the intended task (even if with no-frills). All the same, is it not a waste of time for active developers to be doing testing when they could be coding instead?

The criteria for being an admin is more about competence in communication than anything else:
- Can he/she extract info from the developer about how-to?
- Can he/she explain clearly what isn't working in a way that can be addressed be a developer (i.e. write a proper bug-report and separate one feature from another)?
- Does he/she want to bring as much of the software in the repositories up to release standard?
>> the only proviso is that the tester is aware of what the programme "should" do. This is more of a pre-requisite for a testing than a testing-admin role, I think?

Regarding how to test and what to test, this is an area that needs to be negotiated by community developers (as a group) and those willing to test. There needs to be a very simple checklist for software to pass, which any user can test against; where a programme fails a check point, the tester simply needs to say doesn't pass [test] and describe why. Where bugs have already been filed on a programme, the testing admins role would be to gather reports into a todo list for development. Much quicker than have a/the developer go through the bugs because the collater wouldn't be trying to solve things while organising them!

There is this programme - perhaps it could be the first item to get sorted and pushed through the system? KISStester Package | KISStester TMO. Its official bug-tracker is empty but the TMO thread is full of references to problems. KISStester would provide a good basis for promotion. Some variation is required for different types f software e.g. GUI/terminal/library/etc.

Aside: there are some programmes in extras that are barely passable in terms of doing what they're supposed to. Perhaps all software in the repos needs to be 'bugtested' and there also needs to be a means of downgrading software from extras to -testing?

The priority/order for pushing through programmes will always be contentious. IMHO, it would be best to get all latent** -devel programmes to -testing. Once that state has been reached, process latent -testing programmes on a popularity basis (i.e. num total downloads).

Concluding suggestion: appeal for testers and testing-administrators: users and fringe-members might like to do/know more about maemo and their devices. So long as how-to information exists, the main pre-requisite is moviation and willingness.

In this way, there could be a collection of small teams, each with a testing admin, and a couple of testers. These teams would each have a section of the repositories to bug test and promote. For those programmes that failed the test a bug summary could be posted.

To actually fix bugs, very clever people (developers) are required, who are prepared to edit others' code. For this purpose, might a number of community developers might be happy to act as a pool? The idea being that testing-admins could present a todo list for one of these forgotten programmes to the pool, and the next available programmer can convert a well described bug to either fixed or wontfix. I know bug-fixing can be dull but if you're a programmer it's better than writing instructions, right?

*xTZ: any specified time zone - whatever is specified but the same for all.
**latent - not moving through the repos - e.g. un touched for 4+ months.

This is true to a point. The matter of battery consumption is very difficult to define. Can scripts be developed to do the non-trivial/non-obvious things? I mentioned KISStester above, perhaps appending scripts to this? Manually testing of some things may require a better knowledge of the inner workings of the software than the initial developer, which isn't going to grow the pool of testers.

To my mind, there's also difference between a tester and a testing admin. As I've tried to describe above, the testing-admin needs to be able to bring outstanding bugs together and ensure testing/fixing has taken place. Whilst one person can perform testing and admin roles, I'm not convinced the person signing off a peice software needs to be acutely aware of its inner workings so long as it does what it's supposed to and interaction is either intuitive (think grandparents) or described?

I also believe that a testing admin who is not too aware of programming is likely to have a lower threashold of acceptable hence when to promote, resulting in a more working piece of software (obviously the low threashold is a double-edged sword!).

// Edit:

meant say - I'like to volunteer to contribute to this revamped testing effort that aims to get all & sundry from -devel.to full extras. My skills are native english(en-gb) speaker. I can read C/C++ (& edit badly), as well as other languages with C-style structure and syntacx. Been Maemo-ised nearly a year, well and truly! Also, I'd like a way into developing, eventually so by testing is good.