Community Testing Squad (Sprint task:10.01-02)
 

Hi,

along with the improvements in the package interface[1], we'll create a testing squad, in order kick start the testing process and provide better SW to the Maemo users.

Here is a initial proposal, suggestions welcome:

* Roles:
- Tester - Any community member
- Master/Admin - Selected testing squad members.

There will be some special permissions in the testing interface, these should be only used in extreme cases after prior note sent to the testing squad mailing list.

* Master/Admin's special permissions:
- Can demote packages when there is known blockers. (maintainer obligation)
- Can promote packages when they are stuck in the testing queue for a while without any known blocker.

* Testing Squad mailing list:
- Public mailing list where are discussed any situation/issue/doubts concerning the applications in the maemo.org repositories.
- Receives a automatic notification for each packages that enters testing, is demoted or is promoted.

* Organized testing:
- In each week one of the testing squad members is responsible for the elaboration of a small list of apps that should be tested. We'll start with a 5 a day (or 3 a day) approach in the beginning. Of course the testers are free to test other apps or do all the testing in just one day. The objective is have the apps tested by the end of the week.

After gathering some feedback, I'll post instructions how to join the team.

What do you think ?

[1] - http://wiki.maemo.org/Extras-testing...A_Improvements

Re: Community Testing Squad (Sprint task:10.01-02)
 

I think this is an excellent idea and will hopefully solve much of the frustration people (both devs and users) are feeling over some of the ways that the testing queue functions. :) Well done to the Council for coming up with something so robust.

A small question, is there room in this for people like me... an occasional user-tester? I have neither time nor expertise to do a lot of testing, but make a point of have two or three apps from either extras-testing or extras-devel. I choose ones that I shall actually use so that over the course of a couple of weeks I can make a decent evaluation. I tend to give feedback in t.m.o. threads, and then vote when appropriate.

Will there still be space for people like me to vote?

Re: Community Testing Squad (Sprint task:10.01-02)
 

Great step foward.

Along with this, the definition of Blockers, especially with respect to the requirement of Bug-trackers needs to be clarified in the wiki. Is a bug-tracker required for all packages? or only for large ones? This requirement is openly ignored in many cases and the presence of a testing admin with the ability to demote a packege will help enforce the rules.

I also didn't see any changes to the quarantine period in you suggestions. Reducing the quarantine for updates would bode well among the developers who feel that the current system does not encourage quick bugfixes.

Re: Community Testing Squad (Sprint task:10.01-02)
 

If I ever get my hands on an N900 I'd be all over this. Testing things is what makes this stuff fun! :D.

Looks like good progress in getting software moved from the Devel to the more friendly Extras repository. Excellent job!

Archbyte, regarding the quarantine - in the OP they mention the Master/Admins can promote packages. Assume a developer could contact one of them for brief, quick testing of the app before it got promoted as a "fix" or some such.. maybe? (just a thought)

Re: Community Testing Squad (Sprint task:10.01-02)
 

definitely gonna need a maemo-testers list.

Re: Community Testing Squad (Sprint task:10.01-02)
 

@fatalsaint, I don't think the idea is to have the admins override quarantine limits. The admins would be similar to a communtiy tester but with a few more rights to override an invalid rating and of course the promotion/demotion aspect..

And IMO it would be a good idea to avoid situations of having developers make special requests directly for their apps to 'cut the line' so to speak. This would avoid potential conflict of interest situations and help the admins determine the priorities.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Isn't that the Tester role? I read "Tester" as just a title (i.e. standard use-case based design) to describe what anyone can currently do: go in and vote on an app.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Good point. I can see how this would be an unwanted scenario. How would you determine whether something should skip the test phase/quarantine limit though? Any modification to source code can cause some serious problems... even a critical bug fix. You could wind up with a critical bug "fix" that produces a brick.

Well.. technically I guess that "fixes" the original problem, right :D.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Great news! :)

But am I to understand that once the tests are complete, it will be promoted? Or will there still be the 10 quarantine?

Re: Community Testing Squad (Sprint task:10.01-02)
 

I'd add that when an application is demoted an email should also be sent directly to the maintainer.

Even just an automated email, saying your application has been demoted please check the testing squad mailing list for details.