Re: More testers needed to push apps out of extras-testing
 

I wrote about the process on the developer list in 1st of November. One of the issues was a missing manual QA/testing checklist and that everyone wouldn't need to test everything. Automated testing is then a bonus on top of this (and remove at least #1 and #6). As the testing interface is still the same, there is a queue, and it is not clear who has tested what - I made one text based checklist:

Testing checklist:
1. [ ] Bug database exist.
2. [ ] Licensing ok.
3. [ ] Working provided features.
4. [ ] No missing announced features.
5. [ ] Optification ok.
6. [ ] No performance problems.
7. [ ] No power management issues.
8. [ ] No illegal/dubious content.
9. [ ] No known security risks.

* Copy&paste this to the comment box.
* Put [x] for those tests you have done, elaborate on separate row if the test is FAIL.
* Vote up if there were no FAILs, if there was even one FAIL vote down.
* UI usability issues cannot be used as reason for vote down.
* Always test functionality - that is, run the program and try if it works as it should.

imaginary example:
1. [x] Bug database exist.
2. [ ] Licensing ok.
3. [x] Working provided features.
FAIL: There is choice between tabs and spaces as separators but spaces are always used (see bug: http://url/123).
FAIL: When exporting file the program crashes (see bug: http://url/456)
4. [ ] No missing announced features.
5. [x] Optification ok.
FAIL: the package uses 1512kb from root.
6. [ ] No performance problems.
7. [ ] No power management issues.
8. [ ] No illegal/dubious content.
9. [ ] No known security risks.

Re: More testers needed to push apps out of extras-testing
 

I added the list to the end of http://wiki.maemo.org/Extras-testing/QA_checklist - maybe I'll be then actually used.. :)

Re: More testers needed to push apps out of extras-testing
 

Thanks, that's a nice concise checklist for -testing and a nice addition to that wiki page. However, he was asking about tests that have passed before the app shows up in -testing, ie the automated tests. I'm not sure what they are at present.

Anyone who is going to start working on the wiki, please note my previous post so you don't reinvent the wheel.

One of the biggest problems we face is that people just don't read before they ask questions or even add new wiki pages. That means we end up with a lot of wasted effort and redundant content. Case in point: asking what optified means after i described it clearly a couple pages up. That innocent but already-answered question then took a few more posts that needn't have been made...

Re: More testers needed to push apps out of extras-testing
 

I know I am going to regret sticking my head above the parapet like this but....

Is there anyone or group overseeing this section of the wiki at present? I know that there has been a lot of effort gone into UX and documentation recently so if anyone knows then please shout out. I'm happy to put some leg work into collating things and try to put some words together but don't want to duplicate effort or step on anyone's toes.

Regarding sections in the wiki and pages can we gather suggestions here or should we start a new thread for that?

Re: More testers needed to push apps out of extras-testing
 

Hmm... a list of what the Automated testing covers will be good for n00bs such as myself, I agree.

Anyway, once my situation is clearer (job hunting right now so it is a bit hard to focus on all this) you can count me in, also (this sort of stuff has been my job for awhile so I'll do whatever I can to help :) ).

Re: More testers needed to push apps out of extras-testing
 

Would it be a possibility to get a group of people who's main testing responsibility is to check for optification.
When a new package is released to testing they check it, and post a comment saying if it's optified or not. This would save other people from having to do it.

Or at least we should encourage the first person testing to post a comment saying if the package is optified or not

Re: More testers needed to push apps out of extras-testing
 

From the wiki: https://wiki.maemo.org/Extras-devel#Promotion_checks

Short summary in user terms: It checks that the package can be installed without requiring adding additional repos, plus some copyright and naming stuff. Very basic QA right now, in other words.

Re: More testers needed to push apps out of extras-testing
 

Why not force optification...

Put everything in opt and symlink stuff into / if it is REALLY necessary (.

What about /home/opt/maemo/{bin,libs,share} is libtool run on these folders after installing a package. If so, all stuff could go here and optification could be forced.

I just check ${PATH} and /home/opt/bin is not in it.

if /home/opt/maemo/${libs,shared} is not handled by libtool, we might should LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/home/opt/maemo/libs"

Just some thoughts...

A maemo filesystem guide for developers would be greate. Where to put stuff..

Cheers
Bjoern

Re: More testers needed to push apps out of extras-testing
 

would it be possible to have an app tester application?

for example you can tell it to take a preinstall system snapshot eg memory and cpu usage, directories. could it also have a hook for the application installer to log all the install activity eg file locations. a task manager which can filter the application process like process monitor for windows giving you all the cpu and memory usage. having it log the data. all this data would make more sense to the people who develop the apps then the average user who wants to help. command line stuff is all well and good but people like simple with the ability to go advanced.

One thing that i think is important for app testing is the app enviorment details. what other apps are running app conflicts. Having some app to records this information and compress it to a nice zip file to upload would make bug tracking alot easier. could even write a parser for the bug tracking data you receive to check for consistant data like opt and rootfs usage.

sorry i'm not great at explaining whats in my head.

Re: More testers needed to push apps out of extras-testing
 

@smegheadz, this is a great idea and has been used successfully on windows software projects in my experience. depending on what is in your snapshot (registry, config files, installed files/locations/versions) it can be a very intensive and large output to then diff. I am speaking in windows terms, i know, as i'm still learning *nix and how its all wired.

I would envision something that that would check the above and the other environment variables (existing installed apps/versions/etc, memory footprint, CPU, etc) and then log it to a txt or xml file. then you can run the tool before and after a test app install or use and then copy it off to a desktop for analysis/diffing.

I'd be happy to help test such an app or work with any devs that are trying to plan and implement such.

However i would assume that Nokia/Maemo would already have something similar already in-house. Anybody on the 'inside' care to comment or share on what the process is for how the various quality gates are met prior to code making it to the Ovi store or any other 'official' repository?