Extras-testing QA Checklist and Quarantine Period
 

http://wiki.maemo.org/Extras-testing/QA_Checklist

Feedback please. This checklist is common for developers (before promnoting their applications to extras-testing) and betatesters (before giving them farewell to reach Extras).

It would be good to have more specific instructions to measure performace and power management, since sometimes the problems are not obvious specially in only one session or even a couple of days of casual testing.

Re: extras-testing QA Checklist
 

http://wiki.maemo.org/Extras-testing

Are the promotion/demotion criteria shown there up-to-date (10 days, karma >=10) ? At least I can't see voting down an app actually resulting in a karma loss of 4. The pages also seem to be a bit redundant.

Re: extras-testing QA Checklist
 

I just want to add the suggestion to use top in Xterm or maybe loadapplet as an easy way to see if an app has gone off the rails and is eating excessive CPU/battery.

Re: extras-testing QA Checklist
 

Sure the pages are redundant as I had to leave the office right after posting the URL page here. :) I will continue tomorrow.

But please zerojay & co, mae your aditions directly to the wiki page. Thanks!

Re: extras-testing QA Checklist
 

What is the meaning of this item:

Does this mean that programs such as Octave are unacceptable for extras?

Any program that uses enough RAM to swap out system components will cause noticeable harm to respnsiveness, unless you make sure all UI related executables (and their libraries and data pages) are locked in RAM.


There is also:
I think there should be an exception for CLI programs.

Re: extras-testing QA Checklist
 

There must be a way to discern intended from buggy compromises in performance.

One thing is when one app of dedicated use takes a lot of CPU to perform an action understood and desired by the user (render a 3D graph) and another is to have some widget scrolling in such a buggy way that the system becomes sluggish.

And no matter what, Octave just paused in some inactive window should let the system work as usual, 5 minutes same as 5 days after booting the program.

Icon optional for CLI programs sounds reasonable.

Re: extras-testing QA Checklist
 

I think we should provide a standard icon representing the command line that can be used by all CLI applications. Something simple like a zoomed out black screen with a command prompt, for example.

Re: extras-testing QA Checklist
 

I had forgotten the optification. Please check: http://wiki.maemo.org/Extras-testing...root_partition

Can someone explain in that section how to look whether an installed package is optified or not?

Re: extras-testing QA Checklist
 

Tero (I think) started dumping his thoughts at http://wiki.maemo.org/Talk:Extras-testing already - would be good to merge them.

Re: extras-testing QA Checklist
 

Added a section about non-blockers based on comments read from QA testers these days.

http://wiki.maemo.org/Extras-testing...t#Non-blockers

Re: extras-testing QA Checklist
 

While I do not question in anyway the requiremnt to prevent uncessary power consumption, I also have the feeling to the power consumption criteria needs a more detailed and precise formulation.

DiskUsage for example polls file system size around every 5 seconds and also does graphical refreshes. I assume that this should be avoided if the application is not active and thus can be seen as blocker. From the developer view however I need a clear definition of "active" in this case. I remember a discussion exactly about this topic in the developer mailing list. I havn't yet time to check if there is now a documentation about that (that does not focus on use Gtk and it will make it right) that possibly should even be linked in this case.

WifiInfo on the other hand also polls network information in the background. user feedback for the N810 showed that battery will last around 2-3 hours in such mode, so polling should be avoided for a deactive application, too, and current behaviour is likely a blocker, too. On the other hand there is a new wardriving mode, that creates sounds if a open network is detected and it is intended to have this on even while the application is not active. So battery drain is a also intended consequence. I hope this will not stop the application from getting into extras in future. I also have no problem to warn the user once about this mode using a module dialog or similar. I just would like to have a clearification of such handling before I upload it to extras-testing and get lost in various upload iterations and discussions :-)

Since power management is a very difficult topic also for the devloper I would suggest to add as much cross reference regarding testing on my own and technical solutions to definition of "active" and similar, perhaps build a wiki page on its own for this topic (which did not exist the last time I looked).

Gruß...Tim

Re: extras-testing QA Checklist
 

Definitely, but you know a lot more than me so please edit the wiki page providing more details. Without fear.

I have been editing the neighbor pages and now the should be no overlaps:

http://wiki.maemo.org/Extras
http://wiki.maemo.org/Extras-testing
http://wiki.maemo.org/Extras-testing/QA_Checklist
http://wiki.maemo.org/Extras-devel

Please have a look to these pages since there are many details needing a bit more tuning.

Re: extras-testing QA Checklist
 

My thoughts, reposted from maemo-developers:

Quim, are you still working on these/have a plan or are you happy for someone else to dive in and start refactoring? (Not that it'd be me necessarily, busy weekend and software development is currently taking priority)

Re: extras-testing QA Checklist
 

Any help is welcomed. I probably won't touch it during the weekend.

Re: extras-testing QA Checklist
 

Thanks for the reminder!
I put in stubs for testing tools and started waiting for the test tools documentation to come to the wiki. Then got interrupted with something else and put ideas to the talk: page.

Re: extras-testing QA Checklist
 

I've been using powertop and strace to check CPU activity (power usage).

http://www.lesswatts.org/projects/powertop/powertop.php
http://wiki.maemo.org/Documentation/.../maemo5/strace

Re: extras-testing QA Checklist
 

The QA Checklist is basically a checklist now, as per Jaffa's request. The rest of the beef has been moved to the Extras-testing page.

I have started adding the very basic hints and pointers to each blocker in the QA Checklist in order to help developers and testers evaluating the quality of the software. I'm not a developer and not even a good tester myself, so please help adding useful details *there*. Thank you!

Re: extras-testing QA Checklist
 

Trying to open up the discussion about the rules with the actual developers who'll have to meet them:

http://lists.maemo.org/pipermail/mae...er/021797.html

Hopefully my handy set of bullet points suitably captures the intent of the QA list (and I agree with Attila's points too; "risk" is a vague word ;-))

Re: extras-testing QA Checklist
 

Hm, I just noticed an interesting loophole. So, while "free" packages in extras-testing have to wait for at least x days and x karma, "non-free" ones can skip the whole QA process and go straight to extras. Is it just me or is something very wrong here?

Re: extras-testing QA Checklist
 

Note that those instructions haven't been updated since Diablo. I'd say the non-free apps also go to the extras-testing QA queue.

http://repository.maemo.org/extras-t...ntle/non-free/

Seriously! Extras-testing procedures stink
 

Before anyone complains that there are other similar threads, I don't really care, because the the procedures are still what they are. That is, ridiculous, to put it mildly.

Why on earth does the package loose all the votes and nice comments when a new version is uploaded? Not to mention the resetting of the guarantee time back to 10 days! How could this possible encourage to release often and soon? Yeah, a devs could always release often, but the apps would not be publicly available very soon. More like never!

And what is the point of this general 10 days for all apps, no matter how many lines of code? Yes, for a huge office app that does everything from word processing to spreadsheet 10 days is not very long at all, even too short. But if all the app does is to calculate 2 + 2, why the 10 days? So that some unforeseen errors could be spotted? Like what?! Instead of 4 it calculates 5? Oh dear! :eek: This is bad! :rolleyes: I hope I'm not too presumptuous as to suggest that an error like that could be spotted in less than 10 days. Who knows, there might be some testers that would only need 9. :D

OK, I'm a venting my frustrations here, I admit. :o But the current system is a bit annoying, I think. There should be a better one and soon.

For example. The debian packages do include changelog file. Why not include that to the testing page, so that whenever a new version is updated, the tester could see what has been changed and what needs to be retested. Let us not forget that often the update is a result of the testers feedback. So it would be beneficial if the previous comments would remain, in order to see what was suggested and how it was implemented in the new version.

Obviously, whenever code has been changed, new unforeseen bugs can emerge. Therefore some further testing is warranted. But it should not start from zero again, especially if it was a very minor update. Perhaps something like two more votes because of the update, could be required.

Anyway, just to give an example: I got this Currency Converter in Extra-testing. The previous version was missing the help files. But with that the situation was this:
http://img37.imageshack.us/img37/8252/thumbse.png

The thumb down was mainly because of the missing help. So I corrected that and now I got this. Also all the nice comments that were there are now lost. It's like this app never exited before.

Cheers!

PS. I hope I haven't offended anyone. I appreciate the hard work you are all doing here. :)

Re: Seriously! Extras-testing procedures stink
 

the old page didn't vanish, perhaps a simple list with all previous versions (and links) could solve this elegantly?

edit:
there is such a thing already, a bit confusing though :)
http://maemo.org/packages/view/currencyconverter/

Re: Seriously! Extras-testing procedures stink
 

I couldnt agree more especially when there is a changelog that could be added with every package. Anyways before discussing this further I'd suggest you try the mailin list for maemo, you are better off getting a more concrete response there.
Im eager to see some changes in the procedure too ;)

good luck

Re: Seriously! Extras-testing procedures stink
 

I still don't understand why this couldn't continue under the other thread. Many of the same points are already raised. Why fragment the discussion?

I'm for moving this.

Re: Seriously! Extras-testing procedures stink
 

Yes, the changelog in every package is really missing..

Re: Seriously! Extras-testing procedures stink
 

Yeah, you're maybe right.. Sorry, in my frustration, I had a momentary lapse of reason. :o

Re: Seriously! Extras-testing procedures stink
 

I think you highlight a very specific problem though - the issue of votes being lost if a small improvement is made.

There ought to be some way of making those votes transferable.

Re: Seriously! Extras-testing procedures stink
 

Before we do that, I *still* think we need to step back and ask why/what we want out of voting, and what is the best method.

I understand Sasler's need for a quick fix, but I'm concerned that the core process is broken. That leaves what to do in the meantime, and maybe we need a single trusted gatekeeper with full control until we develop a better system and process.

Re: Seriously! Extras-testing procedures stink
 

Be happy that you've passed the Cerberus guarding upload access to extras.

Re: Seriously! Extras-testing procedures stink
 

I'm also concerned that the core process is broken. The "quality assurance" hoops that a developer has to jump through to get to Fremantle Extras don't feel like they're assuring quality at all. They just feel like they're slowing the process down. Sometimes that helps quality, sometimes it doesn't.

Honestly, it feels to me like the ridiculous changes to airport security. Are we any safer now that people can't take bottled water onto a flight?

Re: Seriously! Extras-testing procedures stink
 

In addition to what Tex says about the fundamental building blocks being broken.. the other thing to consider is that even a minuscule change to a package could have catastrophic consequences.

One would hope these would be discovered rather quickly and fixed again.. but the previous votes don't necessarily apply once there has been any modification done.

Re: Seriously! Extras-testing procedures stink
 

In my ideal world, you'd break down votes into categories, and need to collect a certain number in each category - UI, stability, usefulness, Finctionality, speed/smoothness, help-file etc. Then if you updated, the votes from the elements unchanged would be transferable. So if you added a help file, you'd keep the votes for the other categories, and need to accrue 2 or 3 for the helpfile.

It would also mean that people could vote for the things they like about an app, while remaining neutral or negative on others.

For example I am tesing an app which I adore for its functionality - but the way it is designed means it will never run as swiftly as a competitor. So how do I vote? In a categorised voting, I could vot up for the IU, stability and functionality (actually, stability is much improved in the lastest update) but ignore the smoothness.


Edit!
I get that. But adding a help file won't change the actual app at all. Yet all votes are lost for doing it.

Re: Seriously! Extras-testing procedures stink
 

Right. After release 1 you have to add regression testing into the mix.

Re: Seriously! Extras-testing procedures stink
 

Standard software QA. ;)

Shouldn't anyway. But that points to the need for categorization driving voting.

Thumbs = fail.

Re: Seriously! Extras-testing procedures stink
 

I agree. I don't think the voting system is really working very well. Some seem to give the thumbs up or down based on personal preferences and not the quality of the program. Perhaps a better system would be a kind of check-list where to tester could give a pass or fail to a specific point on that list, based on his testing. Something like this has already been suggested and I think it should be discussed more.

Re: Seriously! Extras-testing procedures stink
 

Which I think is what I was getting at here. :)

Re: Seriously! Extras-testing procedures stink
 

True.. but there really isn't any (easy) way to tell the difference between (in a computerized way):

And:

Type of thing. Granted yes.. technically you can specifically check for if the only change was added to the subdirectory /usr/share/man.. but that's just one specific example. There are many changes to a package that could cause absolutely no damage.. and yet other changes that do. The ability to code a complete AI to parse the package and tell what could or couldn't be harmful is well beyond what I think anyone wants to put into it.

EDIT: Standard warning to accompany that command: don't run that! ever! It will mutate your device into a radioactive turtle that pretends it knows karate and will raid your fridge for all it's pizza!!! Also.. it just might eat your cat...

Re: Seriously! Extras-testing procedures stink
 

I don't have a cat, though a mutant ninja turtle sounds kind of interesting. I have a Michaelangelo hand puppet somewhere...

I take the point. Maybe we need some skilled 'testing-meisters' to check out changelogs and 'port' the relevent votes when appropriate. Of course, it would need people who could actually understand changelogs.

(So go on, tell me, just between ourselves... what does rm -rf / really do???)

Re: Seriously! Extras-testing procedures stink
 

RM is the obvious command to remove files.

-r switch means recursively (so go beneath subdirectories)

-f means force it.. don't ask me if I want to or not.

/ is your root.

Put it all together:

Remove all files and directories, recursively, from the root down.

In other words: Destroy everything.

ETA: Now.. SOME files will be unable to be removed as the OS is currently using them and will fail.. but MOST files WILL get deleted.. including all configuration files, most drivers, most parts of your kernel... etc. The device will be unusable.

Re: Seriously! Extras-testing procedures stink
 

Thank you for taking me seriously, and explaining something very complex in short words even a bear can understand. I shall avoid running that one. :)