Notices


Reply
Thread Tools
Flandry's Avatar
Posts: 1,559 | Thanked: 1,786 times | Joined on Oct 2009 @ Boston
#1
or half finished, depending on whether you are a pessimist or an optimist, i suppose. Looking at the wiki and other sources of discussion on the Extras-Testing validation process, there are a few things that stand out as missing and in need of implementation.

Before i cover those, though, let's review the purpose of -testing:
  • provide a quarantine time for serious problems with an app to be discovered
  • provide Quality Assurance by requiring testing of basic essential criteria for apps destined for Extras
  • act as a "demilitarized zone" for power users to use and test apps

In short, it should be a way for apps to get thoroughly tested for basic requirements by others before allowing an app in Extras. Here are the needed changes to accomplish this in a way that is less onerous to developers and testers alike:

First, a way to have two tiers of testers and criteria. The description was of a pool of "official" testers required to give some percentage of the total thumbs-up. The importance of this is twofold: it allows for thorough testing of all QA criteria, and it allows for "casual" or fan testers, with a smaller mandate, to do the majority of the tests, thus expediting testing. All that is needed for this is a process for admitting "official" testers, a mechanism in the promoter script to recognize and identify those in the test reporting page, and a revised list of QA criteria for the informal testers. The official testers could use the existing comment mechanism to report what they had checked in-depth to verify all points of the QA checklist were covered.

Second, a reduced burden of testing and quarantine for updates. The present requirement for ten karma and ten days does not encourage "release early, release often".

Note that deputizing ordinary users as testers via the mechanism in the first point would help alleviate this update testing problem, but ten days and ten full testing karma is too much. The current testing QA list is too confusing for ordinary users who want to help, which leaves them frustrated at the delay in releases and with no way to help.
__________________

Unofficial PR1.3/Meego 1.1 FAQ

***
Classic example of arbitrary Nokia decision making. Couldn't just fallback to the no brainer of tagging with lat/lon if network isn't accessible, could you Nokia?
MAME: an arcade in your pocket
Accelemymote: make your accelerometer more joy-ful

Last edited by Flandry; 2010-01-18 at 13:47.
 

The Following 14 Users Say Thank You to Flandry For This Useful Post:
Posts: 53 | Thanked: 49 times | Joined on Jun 2007
#2
Yeap, the testing queue is stuck. The queue is now a popularity contest as I was afraid in fall. It is hard to say what people have actually tested, and I think only few have used QA checklist. Plus the website needs some work. Also I think that testing queue shouldn't be a place where software is brought to find out what bugs there is, software should be release quality and testing queue is for acceptance testing only. If a feature does not work, disable it and release when it is ready.

As I understand the medicine is to:
1) Retain discussion history for a package
2) For new versions there should be changelog
3) Automatically test as many things as possible on the checklist
2) Those not automatically tested should be tested atleast by one tester
4) Functionality should be tested by more than one person. I would still keep 10 people as acceptance level for now
5) Thumb down should be accompanied by a bug report

When this system would work, then would be a time to fine-tune:
- How to make bugfix releases go through the system faster?
- How to clean up the queue from stuff which is not going through?
- How many testers is right amount to make certain that functionality is ok?
 

The Following 9 Users Say Thank You to VRe For This Useful Post:
Posts: 1,418 | Thanked: 1,541 times | Joined on Feb 2008
#3
You should only make promotion form Extras-Testing into Extras harder if there are any real problems with Extras packages. As there are no big problems with Extras packages that I know of at the moment, making promotion harder will not bring any advantage. It will simply prevent packages from reaching Extras, spoiling it for everybody.
 

The Following 6 Users Say Thank You to fms For This Useful Post:
ndi's Avatar
Posts: 2,050 | Thanked: 1,425 times | Joined on Dec 2009 @ Bucharest
#4
I agree with VRe. Also, I think more exposure is a good thing. Few people know this is possible or that it helps or know where to vote and what to test.

I'm thinking mandatory exposure to community, say, reserved forum section/threads? Complete with links?

Also, a checklist template could be used to tag votes, maybe we can see later what was tested and wasn't? Maybe by whom and how so we can find and ask later how it worked?
__________________
N900 dead and Nokia no longer replaces them. Thanks for all the fish.

Keep the forums clean: use "Thanks" button instead of the thank you post.
 

The Following 4 Users Say Thank You to ndi For This Useful Post:
Flandry's Avatar
Posts: 1,559 | Thanked: 1,786 times | Joined on Oct 2009 @ Boston
#5
Hang on a moment guys. I don't really care if extra features are added but i think that is Brainstorm material.

All i am requesting up front is that two things already planned to be part of the process--official testers and reduced requirements for updates--are finished. The system as it exists in a half-finished state is not accomplishing its purpose and at the same time getting in the way.

If i knew better where to make this request, i would, but posting a plea in community seemed like the best bet.
__________________

Unofficial PR1.3/Meego 1.1 FAQ

***
Classic example of arbitrary Nokia decision making. Couldn't just fallback to the no brainer of tagging with lat/lon if network isn't accessible, could you Nokia?
MAME: an arcade in your pocket
Accelemymote: make your accelerometer more joy-ful
 

The Following 4 Users Say Thank You to Flandry For This Useful Post:
qgil's Avatar
Posts: 3,105 | Thanked: 11,075 times | Joined on Jul 2007 @ Mountain View (CA, USA)
#6
Originally Posted by fms View Post
You should only make promotion form Extras-Testing into Extras harder if there are any real problems with Extras packages.
We have got at least 3 problems that I recall in this relatively short period: wallpapers with Nintendo copyrights, Radio FM messing with speakers affecting phone calls and Load applet advertising screencasts that actually won't work, confusing several users.

Actually these three problems were not difficult to find, and they were even commented during the QA process. Perhaps what happens is that the goal is difficult to achieve in practice (one person going through 10 blocker criteria, with the skills it implies). Which leads to even the more riguroous evaluators being soft here and there.

I think the key feature that would make testing more accessible to power users and even fun is to split ratings by blocker criteria, as discussed some time ago. Some users will be happy checking whether the features advertized are in place. Others will follow the steps to check whether the app is optified. If at the end the remaining items are system performance and power managent, normal users will be able to assess at least if they noticed serious/noticeable problems. Of course much better if someone can run the right tools and do proper tests, but in their absence happy testers have chances to lead to happy users.
 

The Following 11 Users Say Thank You to qgil For This Useful Post:
qole's Avatar
Moderator | Posts: 7,109 | Thanked: 8,802 times | Joined on Oct 2007 @ Vancouver, BC, Canada
#7
Honestly, my experience from Diablo is that I can leave a package in Extras-devel for 3 months and get almost no feedback, and then I promote it to Extras and I get three bug reports in a day.

I suspect the same thing is happening in Fremantle. I can't be sure, my apps are stuck in -testing with sufficient karma but I guess not enough quarantine time?

The ten days quarantine thing is the part I don't understand. What exactly are they waiting for? If you get the votes, your package should move onwards.
__________________
qole.org --- twitter --- Easy Debian wiki page
Please don't send me a private message, post to the appropriate thread.
Thank you all for your donations!
 

The Following 7 Users Say Thank You to qole For This Useful Post:
qgil's Avatar
Posts: 3,105 | Thanked: 11,075 times | Joined on Jul 2007 @ Mountain View (CA, USA)
#8
Originally Posted by qole View Post
The ten days quarantine thing is the part I don't understand. What exactly are they waiting for? If you get the votes, your package should move onwards.
I guess we could go for this if at the same time it is agreed that packages can be pulled from Extras back to Extras-testing if problems arise later on.

That is another problem. Packages were pulled only when there were serious problems floating and after contacting personally the maintainers. The Load applet for instance is still there, even if the bug is evident. Still, the responsibility of pull it off (or even ask for it) is not defined and there you have still the app getting more downloads and confusing more users.

And don't get me wrong, I don't blame the developers! Bugs happen, problems happen. This is not about being harder in order to try to prevent problem, but about being more flexible so the problems can be dealt easily when they come.
 

The Following 5 Users Say Thank You to qgil For This Useful Post:
Texrat's Avatar
Posts: 11,700 | Thanked: 10,015 times | Joined on Jun 2006 @ North Texas, USA
#9
Testing attributes need to be built into the Maemo infrastructure IMO. Apps should have the right metadata wrapped around them throughout their development/deployment lifecycle, and there needs to be a Maemo-managed web-based system in place to interact with that metadata. Once that's in place, making sure testing efforts are meaningful becomes simply a matter of exposing and updating the proper App attributes at various points along the lifecycle. Right now there does not appear to be enough detail.

related to this Brainstorm: http://talk.maemo.org/showthread.php?t=38014

(sorry, Flandry, I see this was outside your intended scope)
__________________
Nokia Developer Champion
Different <> Wrong | Listen - Judgment = Progress | People + Trust = Success
My personal site: http://texrat.net

Last edited by Texrat; 2010-01-18 at 18:00.
 

The Following 6 Users Say Thank You to Texrat For This Useful Post:
Posts: 237 | Thanked: 157 times | Joined on Dec 2009 @ San Diego, CA
#10
Originally Posted by qgil View Post
I guess we could go for this if at the same time it is agreed that packages can be pulled from Extras back to Extras-testing if problems arise later on.

That is another problem. Packages were pulled only when there were serious problems floating and after contacting personally the maintainers. The Load applet for instance is still there, even if the bug is evident. Still, the responsibility of pull it off (or even ask for it) is not defined and there you have still the app getting more downloads and confusing more users.

And don't get me wrong, I don't blame the developers! Bugs happen, problems happen. This is not about being harder in order to try to prevent problem, but about being more flexible so the problems can be dealt easily when they come.
I think the quarantine is useful, for instances like the load applet, where the thorough testers can shout: "hey I realize this has enough votes but issue X needs to be resolved in some way before this goes to extras"

Since current voting system seems to lend itself more towards a popularity contest as others have said, perhaps the quarantine period could be used as final acceptance testing by 'official' testers?

I agree that a well defined process for removing a package from extras would be beneficial, but I'm not sure that alone is enough to reduce the quarantine time.
 

The Following 2 Users Say Thank You to go1dfish For This Useful Post:
Reply

Tags
extras-tesing, finishing the job, quality assurance, quarantine, software quality, user testing

Thread Tools

 
Forum Jump


All times are GMT. The time now is 22:18.