Community Testing Squad (Sprint task:10.01-02)
 

Hi,

along with the improvements in the package interface[1], we'll create a testing squad, in order kick start the testing process and provide better SW to the Maemo users.

Here is a initial proposal, suggestions welcome:

* Roles:
- Tester - Any community member
- Master/Admin - Selected testing squad members.

There will be some special permissions in the testing interface, these should be only used in extreme cases after prior note sent to the testing squad mailing list.

* Master/Admin's special permissions:
- Can demote packages when there is known blockers. (maintainer obligation)
- Can promote packages when they are stuck in the testing queue for a while without any known blocker.

* Testing Squad mailing list:
- Public mailing list where are discussed any situation/issue/doubts concerning the applications in the maemo.org repositories.
- Receives a automatic notification for each packages that enters testing, is demoted or is promoted.

* Organized testing:
- In each week one of the testing squad members is responsible for the elaboration of a small list of apps that should be tested. We'll start with a 5 a day (or 3 a day) approach in the beginning. Of course the testers are free to test other apps or do all the testing in just one day. The objective is have the apps tested by the end of the week.

After gathering some feedback, I'll post instructions how to join the team.

What do you think ?

[1] - http://wiki.maemo.org/Extras-testing...A_Improvements

Re: Community Testing Squad (Sprint task:10.01-02)
 

I think this is an excellent idea and will hopefully solve much of the frustration people (both devs and users) are feeling over some of the ways that the testing queue functions. :) Well done to the Council for coming up with something so robust.

A small question, is there room in this for people like me... an occasional user-tester? I have neither time nor expertise to do a lot of testing, but make a point of have two or three apps from either extras-testing or extras-devel. I choose ones that I shall actually use so that over the course of a couple of weeks I can make a decent evaluation. I tend to give feedback in t.m.o. threads, and then vote when appropriate.

Will there still be space for people like me to vote?

Re: Community Testing Squad (Sprint task:10.01-02)
 

Great step foward.

Along with this, the definition of Blockers, especially with respect to the requirement of Bug-trackers needs to be clarified in the wiki. Is a bug-tracker required for all packages? or only for large ones? This requirement is openly ignored in many cases and the presence of a testing admin with the ability to demote a packege will help enforce the rules.

I also didn't see any changes to the quarantine period in you suggestions. Reducing the quarantine for updates would bode well among the developers who feel that the current system does not encourage quick bugfixes.

Re: Community Testing Squad (Sprint task:10.01-02)
 

If I ever get my hands on an N900 I'd be all over this. Testing things is what makes this stuff fun! :D.

Looks like good progress in getting software moved from the Devel to the more friendly Extras repository. Excellent job!

Archbyte, regarding the quarantine - in the OP they mention the Master/Admins can promote packages. Assume a developer could contact one of them for brief, quick testing of the app before it got promoted as a "fix" or some such.. maybe? (just a thought)

Re: Community Testing Squad (Sprint task:10.01-02)
 

definitely gonna need a maemo-testers list.

Re: Community Testing Squad (Sprint task:10.01-02)
 

@fatalsaint, I don't think the idea is to have the admins override quarantine limits. The admins would be similar to a communtiy tester but with a few more rights to override an invalid rating and of course the promotion/demotion aspect..

And IMO it would be a good idea to avoid situations of having developers make special requests directly for their apps to 'cut the line' so to speak. This would avoid potential conflict of interest situations and help the admins determine the priorities.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Isn't that the Tester role? I read "Tester" as just a title (i.e. standard use-case based design) to describe what anyone can currently do: go in and vote on an app.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Good point. I can see how this would be an unwanted scenario. How would you determine whether something should skip the test phase/quarantine limit though? Any modification to source code can cause some serious problems... even a critical bug fix. You could wind up with a critical bug "fix" that produces a brick.

Well.. technically I guess that "fixes" the original problem, right :D.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Great news! :)

But am I to understand that once the tests are complete, it will be promoted? Or will there still be the 10 quarantine?

Re: Community Testing Squad (Sprint task:10.01-02)
 

I'd add that when an application is demoted an email should also be sent directly to the maintainer.

Even just an automated email, saying your application has been demoted please check the testing squad mailing list for details.

Re: Community Testing Squad (Sprint task:10.01-02)
 

If you read carefully the suggestions in the link above you'll notice that the bugtracker field will be automatically checked by the autobuilder, so the testers only need to worry about if the bugtracker is invalid (like bugs.maemo.org only). For small packages/themes a e-mail address is enough.

We'll keep the karma for small modifications as described in the link above, as for the quarantine period I'm against reducing it, the developers have to take in mind that we're no testing slaves :D, nor the regular user likes to to have updates poping out each hour(at least I don't like :) ), so the developer need to consider all of these variables for each release. In case of a major flaw, the admins can step in.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Exactly, anyone is free to join in, test only apps that he/she uses and learn more and more. With willing to learn sooner or later will be a good tester :D.

Re: Community Testing Squad (Sprint task:10.01-02)
 

The plan is to keep quarantine, and by my short experience this is very important, I can't remember who suggested the quarantine period, but that person is a smart & experienced person for sure.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Unless there's some bug in the system, that functionality is already there for some months.

Re: Community Testing Squad (Sprint task:10.01-02)
 

OK. Fair enough. I just wanted to be sure. :)

But did I understand correctly from one of your previous posts that in case of a critical update, the Admin could be contacted for a speedy release? As an (extreme) example:

Re: Community Testing Squad (Sprint task:10.01-02)
 

Is it? Thank you. The way I read it - and Texrat's comment about needing a registered list of testers, made me think you were looking for people with particular skills and time commitment. I have no skills beyond being a user, and my time for this at present is 'the time I would be using the app anyway'. (Roll on Summer!)

But if you can use me, please add me to the list. :)

Re: Community Testing Squad (Sprint task:10.01-02)
 

Exactly, FM Radio was one of these examples, we removed it from Extras but we couldn't remove it from every device out there, the best way to fix these issues globally is releasing a new version ASAP.

Re: Community Testing Squad (Sprint task:10.01-02)
 

The trouble here was that ASAP was not possible due to 10 days of quarantine. ;)

Re: Community Testing Squad (Sprint task:10.01-02)
 

No no-- not a registered list of users, but a mailing list for testing. Sorry.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Hi,

I really do like this suggestions! One thing seems to be important for me:


if a package is promoted the should have to be a list of changes. Is there already a possibility in the publishing interface to put a change log or at least a link to the change log location inside of it?
It would really improve the efficiency of tests if the tester knows where are new features (I know that's not all... :)).

3 or 5 a day sounds very much. Maybe we should classify the apps. A complex application should get 3 complex points a simple new desktop theme maybe only 1 complex point. Than it would be easy to say lets test apps with in sum 5 complex points. What do you think?

This answer also goes out to the community mailing list.

Ciao Uwe

Re: Community Testing Squad (Sprint task:10.01-02)
 

Yes, the package interface will show the changelog available in the package.

Yes, we need to ponder about the complexity of the apps.

Re: Community Testing Squad (Sprint task:10.01-02)
 

As I suggested before, app complexity should drive several things, especially a quarantine period if we continue to have one. Lines of code (LOC) is certainly a vague indicator but useful at a high level. Perhaps type/number of libraries used could be too. We don't have to get too deep into details either-- high level approach should be good enough.

Re: Community Testing Squad (Sprint task:10.01-02)
 

I wonder about some sort of place where people could signify what they are currently testing? To prevent a situation where 30 people test one app and none another. Or am I being overly complex here?

Re: Community Testing Squad (Sprint task:10.01-02)
 

An app testing status page is a must IMO.

Re: Community Testing Squad (Sprint task:10.01-02)
 

I agree that a complexity category would be good. Using lines of codes would be the easiest way to do it. For example, there could be 3 categories:

LOC < X = Simple
LOC > X and < Y = Normal
LOC > Y = Complex

Re: Community Testing Squad (Sprint task:10.01-02)
 

Yes, this is definitely necessary. I would even go so far that the Admins should post a list of the apps that should be tested now. Critical updates would get priority and for the rest there could be a simple FIFO (first in, first out) procedure.

Re: Community Testing Squad (Sprint task:10.01-02)
 

First of all, I'd like to emphasize how important this task is and how much I appreciate Valerio's (et al) efforts. This is just not to misunderstand what I'm saying below, you have my full support and I'll try to be a good member of the bugsquad when the time comes. Let me elaborate:

If you want to test just one app, you should be able to do that, without any hassle, and be a tester. If tester here means something else (as in a tester who is a more devoted tester than an ordinary tester), it should be made clear. We don't want to scare people away with special procedures or by making them think they should do something special to be a 'tester' (i.e. not to loose people to nomenclature, like seen from RevdKathy's post).

The point is that IMHO it's the admin who *decides* what's a blocker based on the testing guidelines (and the actions above are just consequences of that decision). One of the major reasons to have an admin is exactly to filter the feedback so packages would not get invalid thumbs-downs because of misunderstandings.

[ plug ]Not really related, but I use AppWatch with great success for this. [ /plug ]

And this is my main beef. The testing squad should certainly help stuck apps, but should also stay clear of replacing users. If the person in charge of an app does not understand what an app is supposed to do (or interested in it), it's not realistic to expect a good QA process. Organized testings also have the problem that the different testing squad members may be in different time zones, different schedules, etc (for example most of my free time is weekends - that's more than 5 days, and I will hardly be able to do anything during workdays so anything that should be done by Friday is an automatic fail for me).

Stuff that is missed out from my QAmaster proposal altogether (just mentioning it so we are in the clear if this is intentional): the cooperation of tester admins with actual developers. Testing, if issues found, should ALWAYS result in a bug report(s) and, preferably a pointer for the developer how to resolve it. Remember, the goal is not to clean out extras-testing (that's just a consequence), but to help streamline promotion to Extras with a special focus on quality.

Another thing which is probably very hard to implement, so I'll just list it as an opinion. Per user thumbs down IMHO is a bad metric and is shows very little about what has (not) been checked and how much overlap there is among the reports. Not only should we have categories, but THAT is our score/karma. I should not care if there are a 100 thumbs-ups and just one thumb down, if that thumb down is really a thumb down (blocker). So the 'karma' of the app is, in fact, not the number of people who checked it, but the number of categories it has been checked on. This works the other way round, too - it's pointless to have 10 thumbs downs because of a common issue. It should be one thumb down (in the summary), which multiple users can confirm, with a bug report, end of story.

Now, if there were folks who read through this, happy to hear comments. No pressure, no matter what solution we end up with, it can't really be worse than the current situation :)

Re: Community Testing Squad (Sprint task:10.01-02)
 

Yup, we need to clarify that, as I said anyone can do testing, like now, however the testers referred above will belong to a special garage group.

I don't see any problem here, also as I said, the plan is have the applications tested by the end of the week doesn't matter when you do the testing. Of course sometimes we can arrange small meetings, but only in special cases.

Yes of course, thumbs down must be commented by the testers, ideally with a bug report, that's one of the proposed improvements.
I'm not sure if I understood the QAMaster role, do you think someone would have time to a role like these ? Or are you talking about a paid position ?

Re: Community Testing Squad (Sprint task:10.01-02)
 

Ideally, a paid position, if nothing else, because of the seriousness of the task - that way things would not get delayed because someone had no time due to job/school/etc, and people stuck in testing would know they have a person to turn to who is supposed to help them (the idea of testing is not to have a hundred people individually go and hunt around about their showstoppers - if we know what the problem is, we should have a fix in place in hours if we can, not weeks !). But, even if there are no funds or possibilities for such an option, we could think about things like rotating QAmastership (say, monthly or biweekly, from the pool of testing admins/moderators) to avoid burning out and making some breathing space for volunteers. Effectively, of course, what I called QAmaster, you could say is a part of testing squad administratorship, my point being that such activity (helping developers through testing) should be considered part of the QA procedure itself (the paid position helping to secure dedicated time for this). Seriously, if one person can just double the throughput of testing by helping the developers not just to identify problems but solve their issues, it would mean a lot more applications in Extras, motivated developers, no 'calculative promotions', summa summarum, an investment IMHO well worth the money.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Humm, I can see some pros but a lot of cons as well. I'm against a paid position as least for now, since I strongly believe that the community can take care of testing and this money can be spent in other things that we can't do. We already proved that with a broken system and disorganized testing :) .
A paid person will scare away a lot of the current testers.

As for rotating the QAmaster among the testing team, well the proposal includes something similar, but of course that person doesn't have total control, when we have a group of capable persons why put all the responsibility in just one person ?

Re: Community Testing Squad (Sprint task:10.01-02)
 

Just a suggestion, wasn't trying to make an agenda out of the 'paid' aspect of the thing (in fact, that's the least important aspect, responded to it as you mentioned it explicitly).

Why ? (out of curiosity, I don't see why being paid would make any difference if the level of authority is the same). Just trying to understand problems to refine the idea.

I wanted to avoid it being an overburdened position that results in tester fatigue in the end. I see the tester admin or QAmaster position being more of a 'grease' kind (exercising authority only on testing errors or resolving a status quo), rather than a heavy-batted, control-oriented one. If we DO have a group of people who can (on the long run) predictably dedicate an amount of their time and are capable to do this - all the better !

Re: Community Testing Squad (Sprint task:10.01-02)
 

Just thinking..

I'd like to have diff -y view of the project changes and LOC for changeset size. This could be against the last accepted package or the first one put in the testing queue if there is no previous package in extras. This way the changes could be verified (their size and what they are). This could be helpful in two cases:

1) Package in extras, a new version is submitted to extras. Large changes (+new features) -> more testing, small changes -> less testing. Concentrate on what has changed.

2) Package is in testing queue, the maintainer updates the software. Changes can be verified to be small -> no need to reset all test cases to zero.

Re: Community Testing Squad (Sprint task:10.01-02)
 

In the perfect world it shouldn't make any difference, but I've seen voluntary/collaborative 'jobs' turned in paid jobs. When someone has the obligation to do something a large number of people will step away for that, that's in our nature. Of course if the paid person is a good leader, he will minimize the damages and probably can still get a lot of help for his tasks.

Agree with your points.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Fully agree. Having a group of dedicated admins is a good thing to start things off. Who knows, maybe down the road, a paid position would be warranted.

The Testing concept was created by the community and having apps piling up in the queue simply looks bad on maemo.org and discourages developers.

The admins could become the de facto point men for testing related help/questions and perhaps well-positioned for "recruiting"/"mentoring" testers. All that is required is a group of dedicated individuals.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Group created: https://garage.maemo.org/projects/testingsquad/

We're looking for committed members for the testing squad that can spend some time testing stuff, and also leading the testing squad in some weeks(that would be easy :) ). Feel free to request membership at garage.

Those that doesn't feel ready or without enough skills, please keep following the group activities and doing some testing in the easier areas.

More news soon.

Re: Community Testing Squad (Sprint task:10.01-02)
 

By skills, do you mean the ability to identify a solution to the problem (i.e. looking through the code)? Or simply knowing what to do when encountering a bug (reporting, etc.)?

Also, I think that somewhere along the road, we should have a mandatory revision of app descriptions (solution #9: http://maemo.org/community/brainstor...ll_basket-002/), as it would help the newcomers (because Maemo 5 and beyond is supposed to target a larger audience) who are unfamiliar with the said app.

I'm not trying to tarnish developer's reputations (I'm definitely in no position to do such a thing), as they clearly worked hard on their respective projects, but rather to benefit the end-user.

Going along with benefiting the end user, I second this.

Re: Community Testing Squad (Sprint task:10.01-02)
 

Got to say great idea !!!

I have noticed the App's section is getting faster at releasing stable updates and fresh apps.

Nice work :)

Re: Community Testing Squad (Sprint task:10.01-02)
 

For the first option would be very difficult to encounter candidates, we've lot of distinct software built with difference languages and tools.

What I mean, is the skills to test some (or all) of the testing points[1], and as you said, the ability to report any issues encountered. Is not a hard task :)

[1] - http://wiki.maemo.org/Extras-testing/QA_Checklist

Re: Community Testing Squad (Sprint task:10.01-02)
 

I'm not an expert on programming or Maemo but I want to help to improve the apps. How is the process? Some administrator will indicate the apss that I'll test?

Excuse me the poor English...

Re: Community Testing Squad (Sprint task:10.01-02)
 

Yes, we'll publish a list of apps each week, more news soon.