View Full Version : [Under consideration] Developers should get karma based on the relevance of their software
A proposal just filed to the Maemo Brainstorm (http://maemo.org/community/brainstorm/):
Developers should get karma based on the relevance of their software (http://maemo.org/community/brainstorm/view/developers_should_get_karma_based_on_the_relevance _of_their_software/)
Currently the developers receive karma basically based on how many Garage projects are maintaining or involved with. This is actually a not very relevant statistic: one developer might have only one project making happy to 100.000 users (see Mplayer) while other might have opened 14 garage projects for 14 command-line direct ports of Debian that actually nobody uses or could care less about.
This is not easy to address as there are many possibilities to be unfair e.g. several developers in a project with different degree of involvement, apps downloaded by thousands that are "easy" ports of projects developed hardly by someone else. I wouldn't make big fuzz of this karma for developers, but it would be good to take it into account somehow.
Currently it's easier to get plenty of karma blogging about apps, talking about apps and commenting bugs in apps... but at the end are the dvelopers who carry with a lot of the hard work. Let's praise them!
Please rate the solutions proposed in the Maemo Brainstorm, add your own solutions and comment (in the Brainstorm if possible).
Ooo, I love contextual rating solutions! Very nice, Quim!
Wouldn't Downloads and comments on apps be good metrics to start with?
EDIT: I'll take that question to the Brainstorm.
attila77
2009-09-10, 20:30
The trouble with download metrics is that you have no feedback how many people UN-installed the app :) Imagine you have an app that has 1.000 downloads and a 4.9 score, and one with 10.000 downloads and a 3.9 score... How would you compare the karma of the two ? Are the extra downloads results of quality and recommendations, or just flashy screenshots and catchy project descriptions ?
I'll write here, better no pollution in the Brainstorm page.
@Quim: I don't see an edit button anywhere and I only have Create solution and idea in the drop-down menu. How do I edit my solution?
@Texrat: They're counted, but very slow :)
The trouble with download metrics is that you have no feedback how many people UN-installed the app :) Imagine you have an app that has 1.000 downloads and a 4.9 score, and one with 10.000 downloads and a 3.9 score... How would you compare the karma of the two ? Are the extra downloads results of quality and recommendations, or just flashy screenshots and catchy project descriptions ?
I have solutions to that... but should we move the discussion to the Brainstorm, or would people prefer answering here as well?
penguinbait
2009-09-10, 20:42
Hmmm, so once I ahve enough karma I can vote and run for council, what other reason should I care about Karma?
Why do I care if my Karma is 350 or 3500, or 35000???
sjgadsby
2009-09-10, 20:52
How do I edit my solution?
For editing, there should be a "notepad with pencil" icon next to your solution. I think.
If you absolutely cannot edit your solution, post what you want it to say here. I can update it. (I think.)
I will write a Karmasutra app to boost my dev karma :-)
In earnest: it will be extremely difficult to find a solution for that. The maemo dev community is still very small (just subtract all the somehow-Nokia-related devs in garage/downloads and see what remains!) which makes me think that it is currently too early to introduce more granularity / more complex rules into the karma formula.
In fact, each app available in downloads should just get huge karma just for trying and getting there! (and additional points should be given for apps ending up in the maemo extras repository).
Also, the state of many apps in garage is quite "open"...
In addition, how should we count the extremely high differences in application complexity? Examples: Canola vs. some (of my) Desktop Search Plugins vs. (future Maemo 5) WRT widgets.
I'd say: let it rest for another 6 months.
For editing, there should be a "notepad with pencil" icon next to your solution. I think.
If you absolutely cannot edit your solution, post what you want it to say here. I can update it. (I think.)
No notepad or pencil (or both) :)
I just want to remove the excessive "Solution #4" in the title.
I confirm Kamen's experience-- I never see that notepad either.
attila77
2009-09-10, 21:32
Yep, I had that problem too. Either we're all dead blind, not seeing the forest from the trees or it's really missing. Considering this is a recurring thing, I filed a bug, hopefully it'll make this clearer...
Look, this is another one of those fuzzy things that tends to get pulled into polar corners. It shouldn't be.
In principle Quim's point is highly valid, and I doubt there would be much disagreement overall. The doubt and discomfort lie in the details.
The best mindset when approaching a fuzzy measuring challenge is to accept at the start that it will be imperfect, and you are looking for the least imperfect solution(s) without expending unreasonable effort.
To that extent we start by recognizing the obvious metrics, determining a useful algorithm and mitigating errors and unknowns.
One unknown is uninstalls. Ok, fair enough. But surely this unknown is mitigated by the fact that ratings tend to be applied after a download. And utlimately almost every app will be uninstalled, updated, reconfigured, etc so this is an acceptable, common risk not isolated to just a few apps but rather spread out over many.
Downloads are a very useful metric. So are ratings. But the point was made that a highly-rated app may have fewer downloads than one rated a bit lower. Fine-- then we weight them accordingly and/or introduce factors into the algorithm that accomodate this. Nothing new here; measurement systems encounter this all the time.
Statistics will tell you that reasonable perfection is attained at about 95% of your goal-- any further and you enter the realm of diminishing returns.
Anyway I think this is achievable... but maybe it does require more thought experiments...
I added a solution (typo in title and formatting didn't work (do I need to put <p></p> or <br/> tags?), can't find option to edit, please fix):
Combination of ratings and distribution among developers but per version
This proposed solution is combining solutions #2 and #3 but factoring in versioning.
The aim is for better apps, so if the developers themselves tell end-users to rate their apps as well as provide feedback, ratings and comments becomes then help improve the apps.
When rating apps, it should be by version (major release only). A version 1.0 of an app can get 3 stars but on the next major version release, say 2.0, it can get 5 stars.
As for distribution of karma, it should be by version (major release only) too. Version 1.0 karma can have 3 developers sharing, but for version 2.0, there can be 5 developers sharing.
I'm a good example of karma-looser :D
I slightly disagree with "version" metrics Reggie proposed above, cause, for example, in our OMWeather project versions are 0.xx all along just because they started like this from the beginning and they are basically for developer and in less part for user (just to check if you have the newest version).
Maybe we can count update rate?
A proposal just filed to the Maemo Brainstorm (http://maemo.org/community/brainstorm/):
Developers should get karma based on the relevance of their software (http://maemo.org/community/brainstorm/view/developers_should_get_karma_based_on_the_relevance _of_their_software/)
Currently the developers receive karma basically based on how many Garage projects are maintaining or involved with. This is actually a not very relevant statistic: one developer might have only one project making happy to 100.000 users (see Mplayer) while other might have opened 14 garage projects for 14 command-line direct ports of Debian that actually nobody uses or could care less about.
This is not easy to address as there are many possibilities to be unfair e.g. several developers in a project with different degree of involvement, apps downloaded by thousands that are "easy" ports of projects developed hardly by someone else. I wouldn't make big fuzz of this karma for developers, but it would be good to take it into account somehow.
Currently it's easier to get plenty of karma blogging about apps, talking about apps and commenting bugs in apps... but at the end are the dvelopers who carry with a lot of the hard work. Let's praise them!
Please rate the solutions proposed in the Maemo Brainstorm, add your own solutions and comment (in the Brainstorm if possible).
What does karma "do" for the developer or application?
Does it make Nokia support and maintain the application or will Nokia disregard these applications just like the ones they delivered with the device?
I'm a good example of karma-looser :D
I slightly disagree with "version" metrics Reggie proposed above, cause, for example, in our OMWeather project versions are 0.xx all along just because they started like this from the beginning and they are basically for developer and in less part for user (just to check if you have the newest version).
Maybe we can count update rate?
I agree -- there should be a tag on the version if it's a major release, minor, bug fix, etc.. That way you can tag v0.0105 and v0.0209 as major releases if you want to.
What does karma "do" for the developer or application?
Does it make Nokia support and maintain the application or will Nokia disregard these applications just like the ones they delivered with the device?
Well, basicaly, it's like "speciallist rating". You can ask a person with high karma something and you'll get your answer/help for sure. It's a community thing, not Nokia or IBM :)
Reggie, that's a great start to providing useful context. Post-beta and mature releases should, IMO, enjoy additional karma... but are we dependent on an honor system here, or is versioning strictly enforced?
What does karma "do" for the developer or application?
Does it make Nokia support and maintain the application or will Nokia disregard these applications just like the ones they delivered with the device?
It would be nice to avoid provocation... I don't see it as productive.
It would be nice to avoid provocation... I don't see it as productive.
Is that;
a) "If you don't have anything nice to say then don't say anything"
or
b) "It's a community as long as you agree and like it - if you don't shut up".
Just curious.
Well, basicaly, it's like "speciallist rating". You can ask a person with high karma something and you'll get your answer/help for sure. It's a community thing, not Nokia or IBM :)
Thanks, I appreciate the clarification.
Reggie, that's a great start to providing useful context. Post-beta and mature releases should, IMO, enjoy additional karma... but are we dependent on an honor system here, or is versioning strictly enforced?
Thanks. See my reply to wazd above: http://talk.maemo.org/showpost.php?p=325090&postcount=16
Is that;
a) "If you don't have anything nice to say then don't say anything"
or
b) "It's a community as long as you agree and like it - if you don't shut up".
Just curious.
c) be a reasonable human being.
Thanks
c) be a reasonable human being.
Thanks
I'll take that as 'a'. Thanks.
Is that;
a) "If you don't have anything nice to say then don't say anything"
or
b) "It's a community as long as you agree and like it - if you don't shut up".
Just curious.
It's simply that your opinion of how well Nokia maintains their apps just gets in the way of the topic at hand and doesn't contribute anything useful towards it.
I'll take that as 'a'. Thanks.
Interpret as you wish, but I deliberately discarded A and wrote my C for a specific reason. Different meaning.
Reggie, that's a great start to providing useful context. Post-beta and mature releases should, IMO, enjoy additional karma... but are we dependent on an honor system here, or is versioning strictly enforced?
Well, versioning can be disregarded - every project has maturity in its settings.
P.S. This is also a quote to wazd's post.
It's simply that your opinion of how well Nokia maintains their apps just gets in the way of the topic at hand and doesn't contribute anything useful towards it.
It just that it made my stomach turn a bit when I see a post from qgil (who I BELIEVE is a Nokia employee) starting a post about community developed applications and highlighting fluff like " but at the end are the dvelopers who carry with a lot of the hard work. Let's praise them!" - when at the same time they are unable to even keep their own developed applications updated - and/or come out with new ones.
Now...If Nokia would actually pay these developers (based on some Karma system) then I think that would be a fantastic idea and I'd cheer along.
So to me, it was apt.
It's helpful to separate roles. Assuming Quim as a Nokia employee is directly responsible for unfinished or unsupported official apps is disingenuous and counterproductive.
can we move on?
It's helpful to separate roles. Assuming Quim as a Nokia employee is directly responsible for unfinished or unsupported official apps is disingenuous and counterproductive.
can we move on?
Well, I WAS going to send him a private message asking who/where I can send inquiries to about support for Nokia developed applications...but it turns out that I got this message:
qgil has chosen not to receive private messages or may not be allowed to receive private messages. Therefore you may not send your message to him/her.
A somewhat interesting choice by someone who in his profile says:
Hi! I work in the Maemo SW team advocating for open source. In practice this means I'm responsible of promoting peace, love and intelligence at maemo.org + all our relationships with free software projects and the community in general.
I guess that doesn't extend to private messages.
I guess that doesn't extend to private messages.
It s mean that he prefer also open sourced messages than private one.
Who's going to start the "community members should get karma based on the relevance of their posts" thread? ;)
bummer, in all seriousness I wouldn't be shocked if Quim turned off his private messages so that he wouldn't have to deal with a flood of "can you send me a developer device?!" private messages. Poor guy's probably got an inbox overflowing with requests already.
And now back to the brainstorm at hand.
I do like the idea of mixing 2 and 3 as mentioned earlier. Hmm.
I'm going to have to defend bummers right to ask; why does Quim or Nokia even care what a members karma is? Are there prizes to be won?
I appreciate the push to Maemo5 but at some point someone is going to realize that quite a few of the members/developers/enthusiasts on this board are here for what the N8**'s aspired to be. And that BTW, did not include a dang cell phone.
Now Nokia is asking these same members to pay 60%* more for a device that has features they either don't want or can't use. For many, that price is to high to pay. Particularly when they are already holding a perfectly good device in their hand (an existing tablet) that has yet to reach its full potential.
At the same time these same members are seeing threads telling them that they can "help" Maemo5 by contacting developers and telling them to pour all future efforts into new versions of their apps that won't run on the older devices.
I can see where some might feel left out or left behind. Can anyone else?
*N800 original MSRP = $399
@bummer... Unfortunately that's the way the world goes 'round. However, I believe that we will see a new wave of enthusiasm and member support for the older devices as they are passed down or sold. Take it easy on Quim, he has been good for this community so far and I believe he will continue to be a valuable resource to users of the older devices, as his time permits.
I'm going to have to defend bummers right to ask; why does Quim or Nokia even care what a members karma is? Are there prizes to be won?
I appreciate the push to Maemo5 but at some point someone is going to realize that quite a few of the members/developers/enthusiasts on this board are here for what the N8**'s aspired to be. And that BTW, did not include a dang cell phone.
Now Nokia is asking these same members to pay 60%* more for a device that has features they either don't want or can't use. For many, that price is to high to pay. Particularly when they are already holding a perfectly good device in their hand (an existing tablet) that has yet to reach its full potential.
At the same time these same members are seeing threads telling them that they can "help" Maemo5 by contacting developers and telling them to pour all future efforts into new versions of their apps that won't run on the older devices.
I can see where some might feel left out or left behind. Can anyone else?
*N800 original MSRP = $399
@bummer... Unfortunately that's the way the world goes 'round. However, I believe that we will see a new wave of enthusiasm and member support for the older devices as they are passed down or sold. Take it easy on Quim, he has been good for this community so far and I believe he will continue to be a valuable resource to users of the older devices, as his time permits.
No one is denying his right to ask. This thread just isn't exactly the best time for asking and it's for that reason that I won't go into why a good amount of what you said afterwards is wrong.
If you guys want to discuss it, open a new thread.
Karma is a reputation system that was discussed, implemented and fine tuned within the Maemo community. Like in any reputation system, if you don't like it you can just ignore it, push for improving it or remove it. But this is not the point of this thread.
Karma is one of the elements taken into account in the device programs targetting Maemo contrigbutors. These days we are working on the N900 device program and it's quite evident that developers are ranked with all kinds of distortions and the system seems to praise more the ones talking than the ones coding. Some coders talk a lot, some others don't.
It's too late for any fix now, but it's good to brainstorm for the future. I think we should discuss these ideas thinking on the Maemo 5 timeline, and the implementation should help evaluating the reputation of developers in the near future.
My proposals are not discriminating downloads or evaluations of apps per targeted devices so I don't see the point of discussing something like the N900 price point and the telephony features. There are threads for that already open. Actually the karma system in itself tends to praise the developers that have been here for a long time over the newcomers attracted by the N900, that will start from scratch (but could catch up fast if their apps were great and some of the ideas in this proposal were implemented).
I disabled the private messages in ITT few days after registering, since I don't want to get questions that can be made in public. People that really want to contact me privately can find out pretty easily how to do it. But this is beyond the point of this proposal.
Any other rants you have with Nokia are beyond the point of this proposal. Please get used to see Nokia employees discussing openly about things and keep the discussions on topic no matter what. Creating a new thread linking it from here takes only a little extra effort. Really appreciated.
penguinbait
2009-09-11, 18:15
What about people who provide large amounts of software that is not in the garage. Is there some way I can provide my download numbers from my websites (penguinbait.com / tablethacker.com) and have them count towards karma?
Or is not being in the garage BAD karma :confused:
As long as your software goes through Extras and shows up in maemo.org/downloads, the rest is quite irrelevant.
javispedro
2009-09-11, 19:16
Do we really get karma for "Garage projects"? The http://wiki.maemo.org/Karma page says so, but then does not explain how's it calculated. And today's the first time I've read about it.
If it's true and Garage is a karma source, then I'd vote for it to be removed too. It would only promote the creation of "empty" garage projects; there are quite a few already.
It would only promote the creation of "empty" garage projects; there are quite a few already.
Thanks for reminding me. I've got one myself and I can't seem to figure out how to get rid of it. Help?
penguinbait
2009-09-11, 19:54
As long as your software goes through Extras and shows up in maemo.org/downloads, the rest is quite irrelevant.
Were you answering me? Does this mean the answer is NO?
I assume so, but it was not really clear to me?
javispedro
2009-09-11, 19:56
Thanks for reminding me. I've got one myself and I can't seem to figure out how to get rid of it. Help?
I at least have a "rm" project link in https://garage.maemo.org/my/ . But of course, I only mean totally empty projects.
I at least have a "rm" project link in https://garage.maemo.org/my/ . But of course, I only mean totally empty projects.
The "rm" link wants me to pass admin on the project to someone else. Oh well.
The "rm" link wants me to pass admin on the project to someone else. Oh well.
We can make that a "hot potato" project and just pass it around. :D
smackpotato
2009-09-11, 20:19
Its true people do game the system. I"m currently working on an app to dim the screen when cp cycles used nears 100%. If it wasn't for karma. I'd just keep it for myself. Trivial apps do matter.
What about people who provide large amounts of software that is not in the garage. Is there some way I can provide my download numbers from my websites (penguinbait.com / tablethacker.com) and have them count towards karma?
As long as your software goes through Extras and shows up in maemo.org/downloads, the rest is quite irrelevant.
Quim's answer is more or less accurate. If your software is linked at http://maemo.org/downloads/, you get karma when it's rated & downloaded. tablet-encode (http://mediautils.garage.maemo.org/tablet-encode.html) is not in Extras, but gets me karma. Garage has nothing to do with download karma.
penguinbait
2009-09-11, 23:55
Quim's answer is more or less accurate. If your software is linked at http://maemo.org/downloads/, you get karma when it's rated & downloaded. tablet-encode is not in Extras, but gets me karma. Garage has nothing to do with download karma.
Sorry I thought it was clear, I said the files are posted on penguinbait.com and tablethacker.com. I asked IF there was a way to submit my download numbers to receive karma. So again "yes here is how" or "NO you cannot" is what I was looking for.
Not a reiteration of how it works, that was clear.
So having a repo or website of your own and providing all the software in the world won't allow you to gain any karma. :confused:
I asked IF there was a way to submit my download numbers to receive karma. So again "yes here is how" or "NO you cannot" is what I was looking for.
No, you cannot submit your own download numbers. a) the interface and format are too hideous and complicated to think about; b) although no-one's attempted to really game the system, that'd be far too trivial.
However, if an application is downloaded through http://maemo.org/downloads/, it's count is increased even if the ultimate destination is on another machine (IIRC).
So having a repo or website of your own and providing all the software in the world won't allow you to gain any karma.
Please don't ask questions and then make statements based on an assumed answer.
penguinbait
2009-09-12, 00:39
Please don't ask questions and then make statements based on an assumed answer.
I had to assume what the answer is because initially neither you nor Qium could clearly articulate the answer, although I think you thought you did.
So did I misspeak?, was my assumed answer not correct?
As long as we are asking nicely, Please don't tell me what to do.
However, if an application is downloaded through http://maemo.org/downloads/, it's count is increased
Hm, is that what we want to count for downloads karma? I expect most downloads to happen from h-a-m in which case the repository access logs should be used instead. Maybe someone with access to both sets of stats could run a comparison?
If a developer is not using maemo.org Extras neither maemo.org Downloads then it makes sense that he doesn get maemo.org karma, isn't it?
Hm, is that what we want to count for downloads karma? I expect most downloads to happen from h-a-m in which case the repository access logs should be used instead. Maybe someone with access to both sets of stats could run a comparison?
The downloads from the repository are also counted in maemo.org downloads. They are synchronized every X time. Where X is a value that X-Fade knows ;-)
Hm, is that what we want to count for downloads karma? I expect most downloads to happen from h-a-m in which case the repository access logs should be used instead. Maybe someone with access to both sets of stats could run a comparison?
Download counts are updated by clicks on the install icon(live) and are corrected once a week by the actual download stats we receive from the caching network. So in the end we only count actual downloads from the repository.
We're looking into going from weekly to daily corrections, but that requires some changes and bureaucracy, so that might take a while still.
Another thing we could use is the app karma feature (In Downloads). App karma is basically going to show which apps are relevant. (Relative downloads stats, comments and ratings are being used there)
This app karma can be used as a modifier for the actual karma the developer gets for that app. App karma is a float value between 0 and 1.
The most relevant apps with have an app karma value of very close to 1.
Calculation could be then be:
default karma value per app * app karma.
Example:
default karma per app = 20
karma for my app X -> 20 * 0.342234 = 7 karma
This would also mean that once your app gets less relevant, your karma for it will go down. It will be more dynamic and your karma will fluctuate a bit.
attila77
2009-09-12, 10:19
A random thought on download karma... Merging versions with downloads is a bit tricky, right ? I mean, when you push something to extras, you will certainly see a surge in downloads from people who are updating in addition to those who are installing for the first time. This means projects updating more often will certainly see a higher number of downloads even with the same userbase...
... and now, the kicker...
What if we use these spikes as an additional metrics in some form ? After all, the volume of the spike is the approximate numbers who still have it installed. So it is possible to measure uninstalls, even if in an indirect form. Yes, I know, some people never update, but the point is that the metric will be equally wrong on all projects, so you do get a realistic comparison of userbases. Note that this wont work if you update software TOO often as the update dispersal can take a while.
See this graph from maemo mapper - you see a steady stream of downloads, but the spikes containing the updates are those which show the real userbase increase.
EDIT: I'm not saying this should replace download count, but could be something like the t.m.o. posts vs thanks karma. Download karma (=posts) is cheap, update karma (=thanks) is value.
penguinbait
2009-09-12, 21:52
If a developer is not using maemo.org Extras neither maemo.org Downloads then it makes sense that he doesn get maemo.org karma, isn't it?
Not really, there are several people who work on projects that thousands of people use and attract people to the platform that never get put in the garage. Do these people not deserve some sort of Karma for their efforts? Perhaps its impossible to quantify perhaps its not. You can see here http://logs.penguinbait.com/index.html. I have dedicated my home website to mainly Nokia Tablet related traffic. Additionally my numbers from tablethacker.com which is also dedicated to Nokia Internet Tablets are even greater. So I am just saying should there be a way for people who have continuously supported the platform, drawn many people to the platform, published many packages but never in the garage to receive some sort of karma for their efforts.
I understand the fact that you want things to be put in the Garage and in maemo extras. I feel as though there should be some sort of QA approval process if something is going to be placed into maemo/extra's so users can expect some level of consistency. While I think that "hackers" or "hacks" providing beta software at best should never be placed into maemo/extra's I still think they play an important role in the community.
It does make sense that maemo.or karma is measured against factors maemo.org can register. But if you have additional ideas please submit them in the Brainstorm proposal we are discussing.
Note that I'm not mentioning Garage at all. Host your code wherever you prefer.
We are talking about distribution channels. You have extras-devel, extras-testing and Extras for your software in unstable, testing or stable quality. If you need something else then file an enhancement request or an idea in the Brainstorm proposal that started this thread.
And we are talking about maemo.org/downloads that, true, if you are distributing "hackish" software you better don't use.
You can use bugs.maemo.org and that activity will bring you karma.
Yo can talk here about your software, bed thanks and get karma for it.
You can blog about your software and get thumbs up getting karma as well.
The system is probably not so bad when you have 377 karma http://maemo.org/profile/view/penguinbait/
atila77, your reasoning about downloads is very good! Maybe we need to use those peaks for karma instead of full downloads, indeed. Could you please submit the idea? It seems that we can keep the ratio 100 downloads = 1 karma for these peaks? If v1.0 gets a peak 1000 downloads you get 100 karma points. If v1.1 gets a peak of 1500 you get 150 instead of 100. But if v1.1 gets a peak of only 200 then you get 20 instead of 100. Makes sense.
XFade's point about app karma is also good. We should use it somehow to add karma, but perhaps not to diminish karma. We have talked many times about users diminishing karma by 'being older' if they don't add new karma, to make old disappeared contributors go down the ranking while new contributors very active go up up. It lools that this way would be enough to get minus karma. Just like we do with news, they push your karma up but won't push it down.
If karma app is a value between 0 and 1 then waht about this:
karma points = max app karma * 100
I think nobody has really explained how karma is currently calculated for appliactions (Products in profile page).
You get couple of points (<10) for having an application in maemo.org/downloads, then you get additional points for the rating. Maximum karma for one application is 45 or so. I couldn't find the formula so this is from memory. There is no karma for downloads.
Garage karma is called Groups in profile page. If you have single project where you are the maintainer you get 9 points ( I think it is 3 for having an account, 3 for being member of a project, 3 for being maintainer of a project)
attila77
2009-09-13, 11:37
Could you please submit the idea?
OK, let's just clarify a few things beforehand, as there is no edit.
It seems that we can keep the ratio 100 downloads = 1 karma for these peaks? If v1.0 gets a peak 1000 downloads you get 100 karma points. If v1.1 gets a peak of 1500 you get 150 instead of 100. But if v1.1 gets a peak of only 200 then you get 20 instead of 100. Makes sense.
Not sure if it's just terminology, but I strongly suggest peak volume instead of peak value. It would be a shame to loose karma because your peak was at night (split across two days), influenced by day-of-week, holidays, etc.
BTW, you say 100 : 1 but the ratio is 10 : 1 karma in the examples :)
Can someone clarify the X axis relation in the graphs (Niels) ? I can see the data points are weekly, but I don't know if it means the value is SUM of dowloads that week, just a snapshot for a single day or daily AVG for that week which might influence karma multiplier choice... Or, if it is possible, could I get daily resolution data for a project just to see clearer ?
About updates being too close - this is actually not that big of a problem as I initially thought. It is on the current graphs as you can't differentiate between versions, but since these ARE separate files, they might just as well be statted separately for karma purposes.
Here's another graph, this time for OMWeather - it has updates more often than mapper so you can see the principle even better:
Ok some observations:
1. There is no way to edit your submitted solution in Brainstorm (Bug submitted (https://bugs.maemo.org/show_bug.cgi?id=5138)). There is no 'Brainstorm' component in bugzilla either.
2. The discussion of this brainstorm topic at the Brainstorm page itself has only about 13 replies, here at Talk has reached 7 pages already. I propose using Talk to discuss brainstorm ideas, or at least provide a way to link a brainstorm idea to a Talk thread.
3. I propose prompting for a reason why you are voting 'no' on a proposed solution. I think it is good to gather the 'cons' of a proposed solution to give way to a new proposed solution, or to clarify misunderstandings. Also, it might be good to provide a way to change your vote for a proposed solution from yes to no or no to yes.
Just to comment on peaks. This can be abused as well. A version 1.0 can get a high peak, and v1.0001 can also get a high peak for fixing a bug. v1.0002 can get a high peak again for fixing another bug. A developer can get more karma by spreading bug fixes across different versions.
attila77
2009-09-13, 13:01
Just to comment on peaks. This can be abused as well. A version 1.0 can get a high peak, and v1.0001 can also get a high peak for fixing a bug. v1.0002 can get a high peak again for fixing another bug. A developer can get more karma by spreading bug fixes across different versions.
Err, the way I understood it peak karma is not cumulative, you always get the value for the last peak.
Why are we even discussing changing this metric now, so close to a new device release?
(...and before anyone jumps down my throat I am just playing the devils advocate here.)
To a casual observer it appears that someone doesn't like or think that a member or members who have high karma now, deserve any of the future perks that may come from that metric.
If this is the case, what's going to happen the next time if high karma earners from this change fall out of favor (http://Gerrymandering)?
What I'm saying is that if a future perk, developers device, or all expense paid trip to Las Vegas (release rumors early and update often :D ) or something is going to be determined by karma, changing the equation of how karma is calculated now might seem a bit tacky and disingenuous.
Some may discount my post based on my relatively low karma but I am an observer of the process. :)
I'm proposing to discuss this now in order to have the machinery totally fine with a post-N900 device comes. Yes, we are that slow... :)
penguinbait
2009-09-13, 16:21
Is having some kind of routine to keep track of installed software built into application manager that can report back to maemo.org what is currently installed when it performs an update. Perhaps this is going to far, to much like big brother?
CrashandDie
2009-09-13, 16:38
Is having some kind of routine to keep track of installed software built into application manager that can report back to maemo.org what is currently installed when it performs an update. Perhaps this is going to far, to much like big brother?
http://popcon.debian.org/
This has been done and is available for quite some time.
I couldn't find the formula so this is from memory. There is no karma for downloads.
Here are the formulas in use: http://wiki.maemo.org/Karma
Here are the formulas in use: http://wiki.maemo.org/Karma
I've proposed that there be more links to get to that and similar pages. See bug 4959 (vote and comment please!)
https://bugs.maemo.org/show_bug.cgi?id=4959
Err, the way I understood it peak karma is not cumulative, you always get the value for the last peak.
In that case, that might be more dangerous then. There is a chance that the developer might stop improving an app if the peak is already high, despite a lot of requests to add more features. The result: stale apps.
Another thing that might happen is the developer(s) might release that same new app with new features added and appending a 'pro' or 'ultimate' to the app name. The developers get to preserve the high peak on the old app, and get new karma on the 'pro' and 'ultimate' versions.
I believe in rewarding the developers who continuously improve their apps. If they get a million karma because they keep improving their apps, then good. They deserve it.
attila77
2009-09-14, 01:24
In that case, that might be more dangerous then. There is a chance that the developer might stop improving an app if the peak is already high, despite a lot of requests to add more features. The result: stale apps.
I'm not sure I follow... Someone is afraid to post a new version of his software because he fears more users might have uninstalled than installed it in the meantime ? That almost sounds like a psychological problem...
Another thing that might happen is the developer(s) might release that same new app with new features added and appending a 'pro' or 'ultimate' to the app name. The developers get to preserve the high peak on the old app, and get new karma on the 'pro' and 'ultimate' versions.
Now, if someone wants to abuse the karma system, he will do so, you can't stop karma wh*ring, no matter what system you come up with, especially if the karma system is public and the users are anonymous. The whole thread started off because now we can't tell important projects from unimportant ones. Not to mention there are a LOT simpler ways to fake karma than such constructs.
I believe in rewarding the developers who continuously improve their apps. If they get a million karma because they keep improving their apps, then good. They deserve it.
But this is the problem, what's the non-karma-abusable measure of app improvement, if it exists at all ?
How about project karma has an expiration. After a set time development karma would expire maybe a year. this would stop people from sitting on a project and not updating it because their karma would go away. Equally karma from blogging and helpful posting could also expire. If a user used to supply a great deal of information but after a new device or os upgrade comes out becomes very quiet the high karma currently does not reflect the users activity. Questions directed toward a quiet user based on current karma may not get answered with the same speed or quality of response.
I don't think that karma should be subtracted just expired hopefully this would avert negative karma.
If there is negative karma this to should expire. Maybe after a longer time than positive karma expires. people sometimes do bad things but should have the ability to re-hab and not have bad karma follow forever.
There is a chance that the developer might stop improving an app if the peak is already high, despite a lot of requests to add more features. The result: stale apps.
If a developer's primary motivation is to accumulate karma points then their apps will get stale no matter what the formula used. I don't think there's much risk of that though, as most reasonable developers would value their time much higher than a potential device discount or weekend attending talks in Amsterdam :-)
I would like to see some more quality (as opposed to popularity) metrics used however. User ratings certainly, and also things like ratio of open/fixed bugs, code checkins and so on. Yes, all of this can be "gamed" to an extent, but let's please assume that someone who bothers to climb the steep learning curve, negotiate the autobuilder etc to publish an app does so because of a genuine desire to scratch an itch and contribute. There are far easier ways to get karma points after all.
Here's the thing. I'm suggesting to bring in the end-user to reward the developer(s) so apps keep improving. If you provide a solution that somehow stops the development of the app, even how small, then that's a potential problem.
I like how the site versiontracker works. Users provide feedback for each version, and each version gets to be rated (e.g. this (http://www.versiontracker.com/dyn/moreinfo/macosx/10281562&vid=11006639&mode=info)). You can immediately see that the system works. The only problem is how to incorporate karma to it.
Is there a successful system that rely on peaks?
hhedberg
2009-09-14, 05:57
What if we use these spikes as an additional metrics in some form ? After all, the volume of the spike is the approximate numbers who still have it installed.
There was a scientific paper of this phenomenon in the latest Open Source Systems conference (OSS 2009). Here is the reference:
Wiggins, A., Howison, J. & Crowston, K. 2009. Heartbeat: Measuring Active user Base and Potentieal User Interest in FLOSS Projects. In Boldyreff, C., Crowston, K., Lundell, B. & Wasserman, A.I. (eds.): Open Source Ecosystems: Diverse Communities Interacting, Proceedings of 5th IFIP WG 2.13 International Conference on Open SOurce Systems, OSS 2009, Skövde, Sweden, June 2009.
I don't think that karma should be subtracted just expired hopefully this would avert negative karma.
Agreed, and maybe the best way to keep karma current is to constantly calculate it within a moving window, say 1 year. If people cease to be active, their karma will naturally decline... and this also ensures that karma reflects currency.
I would like to see some more quality (as opposed to popularity) metrics used however. User ratings certainly, and also things like ratio of open/fixed bugs, code checkins and so on.
The ratio approach makes the most sense to me and should be used where ever possible and practical. So maybe a download metric incorporates percentage of views converted to downloads? Or maybe the first released version of an app is solely a benchmark, and karma is applied to ratio (and/or number) of updates after initial download? Surely loyalty to a developer or app is an excellent indicator of quality... and addresses the uninstall question to an extent.
benny1967
2009-09-14, 08:38
May I just throw in a number of criteria that may be useful or not. I'm not very good at doing the maths in terms of karma=(a*b/17)^2+(e)*f, and I'm also not sure if i got the idea of what "developer karma" should be... anyway:
Port vs. native development
I once ported a command line application and used it on my device without publishing it. When packaging and extras became an issue (developers told us here that the whole process is broken and oh so difficult), I tried if I could bring it to extras without prior knowledge of debian packaging. I could. An I went the whole way and included it in /downloads/OS2008/. When I asked on the mailing list who I should name as the author of this package, I basically got "whoever you want to get karma for it". Now I wanted to be named as a contact person, I wanted receive karma,... but still: Is it OK to apply the very same developer-karma calculations for someone who'd only re-package an existing tool without even touching the source code? Maybe using two fields in the downloads page ("author" vs. "contact") could be a starting point to change this.
Dependencies
What about packages that are listed as dependencies of other packages? Say I port either a library (or an interpreter for a new language...) to Maemo, and 359 applications start using this library (or interpreter). Most probably nobody would ever download it from /downloads/OS2008/ or comment on it, but without it, 359 high-rated apps wouldn't have been possible. So: Count dependencies in packages.
Rating vs. downloads vs. comments
Relevance is an underdefined term. Whats relevant? Is a game relevant because it has high download rates? Is mplayer relevant because of its download rates... or is it relevant because it pushes the boundaries of the platform? I don't know. - I do believe, though, that counting only the average rating or the number of downloads or the number of comments given for this application will probably give a wrong result.
Downloads can be wrong for a number of reasons already discussed in this thread
Rating is a most irrelevant figure as long as it's the average rating. I give my own application 5 stars and have a higher multiplier than a really popular app that users find minor bugs in and that therefore ends up with 4 stars.
Number of comments could be nice to throw in. How many people think your application is worth writing about? (These could be, of course, all bad reviews like "What's that? It doesn't even install!", so it needs to be just one factor of many)
The penguinbait-effect
There must be a way to gain karma by doing things that don't quite fit the "install a deb from extras" scheme. Many interesting concepts that show how versatile the whole ecosystem is will not, cannot or should not be deployed to end users this way. Still, I'd personally consider them at least as (if not more) relevant than a game that gets downloaded 2000 times a week.
We used to have thumbs up/down on the maemo.org profile pages. This never worked for karma, IIRC, and is now disabled. How about adding a way for developers to add specific projects to their profiles (name of project, link to project homepage, short description, no screenshot) and allow others to give karma via thumbs up?
Fire exit
Karma calculations are one thing and will always be "wrong" for some people, no matter how good the system actually is.
The developer device program is another issue. It can be, but needn't be exclusively tied to karma. You could, say, make it public that a list of x developers will receive a discount based on their karma. Afterwards, those who're not part of this list should be given a chance to be elected by popular outrage ;) ... you know, any developer should be able to stand up and say "hey! community! i think i do deserve it for my projects, but the karma system doesn't work for me. you know me. you love my projects. vote for me at...". we're missing the "at ..." for such a community-ticket.
non-developers
During the N810 device program, I read:
We are running a N810 maemo device program open to current and potential contributors. Note that we have removed the word “developer” to make it clear that non-programmers are included as well.
Is this still valid or will we fall back to developers only again? Just confused because this thread is about developers only. (And no, I'm not asking for myself... I live in a country without a Nokia online store, so I wouldn't be able to use any discount code, anyway.)
attila77
2009-09-14, 09:31
Here's the thing. I'm suggesting to bring in the end-user to reward the developer(s) so apps keep improving. If you provide a solution that somehow stops the development of the app, even how small, then that's a potential problem.
You realize that if you go through with this chain of thought it means development should NOT relate to karma ? If you have karma, you have competitiveness, and if we really want to hair-split that in itself can result in scenarios that stop development (isn't this the crux of the penguinbait conundrum is this very thread ?). As Texrat pointed out, there is no 100% scoring system, the best we can do is come up with something that works in as close to 100% as possible while covering as many scenarios as we can.
Is there a successful system that rely on peaks?
No idea, it was just a random thought (which apparently popped up in other places, too). This changes the question into is there a system that relies on peaks and failed because of it ?
Port vs. native development
Is it OK to apply the very same developer-karma calculations for someone who'd only re-package an existing tool without even touching the source code?
Original code should count for more IMHO, but even straight ports should get something for the packaging and maintenance effort.
Maybe using two fields in the downloads page ("author" vs. "contact") could be a starting point to change this.
I think the "author" field comes from the Maintainer: field in debian/control, and unfortunately there's no such distinction in the Debian spec (http://www.debian.org/doc/debian-policy/ch-controlfields.html) :-(
Another problem is that the distinction is not always clear-cut. We have straight ports (eg most command line tools) and completely original maemo-specific apps at the two extremes, but also ports with lots of maemo-specific changes inbetween, sometimes with the "porter" being the same as the "author" (eg claws-mail) and sometimes a different person (eg xournal or pidgin).
Dependencies
What about packages that are listed as dependencies of other packages? Say I port either a library (or an interpreter for a new language...) to Maemo, and 359 applications start using this library (or interpreter). Most probably nobody would ever download it from /downloads/OS2008/ or comment on it
A library wouldn't even appear there in the first place. Maybe package maintainers could get an additional x points for each package that depends on one of theirs?
Fire exit
Karma calculations are one thing and will always be "wrong" for some people, no matter how good the system actually is.
The developer device program is another issue. It can be, but needn't be exclusively tied to karma. You could, say, make it public that a list of x developers will receive a discount based on their karma. Afterwards, those who're not part of this list should be given a chance to be elected by popular outrage ;)
I agree, and this should also apply to other karma-based rewards like summit sponsorship.
(Thanks for an excellent post btw)
Another thing we could use is the app karma feature (In Downloads). App karma is basically going to show which apps are relevant. (Relative downloads stats, comments and ratings are being used there)
Additionally, we might utilize commit counts from Ohloh (see for example MaemoPlazer (https://www.ohloh.net/p/5850/contributors)) to figure out how much each developer is contributing to the project.
For karma and ohloh see https://bugs.maemo.org/show_bug.cgi?id=5012
This shows that we should move good ideas from bugzilla and talk to brainstorm...
btw Henri, can you have a look to the permissions? People posting ideas can't edit them https://bugs.maemo.org/show_bug.cgi?id=5133
attila77
2009-09-14, 12:54
Additionally, we might utilize commit counts from Ohloh (see for example MaemoPlazer (https://www.ohloh.net/p/5850/contributors)) to figure out how much each developer is contributing to the project.
Commit count is a very unreliable stat (as people have different (D)VCS usage patterns), and quite often newcomers or non-regular contributors just send patches to core developers, who then apply them to trunk.
I would also join the request of those who would like to see a karma multiplier based on just how Maemo a project is, say along the lines of 'straight port', 'localized Maemo application' (for something properly hildonized, etc), 'designed for Maemo' (for something written exclusively or primarily for Maemo).
You realize that if you go through with this chain of thought it means development should NOT relate to karma ?
How's that so? If a small, rewarding karma system was in place during the active Canola development days, iNdT should have gotten so much karma now -- and they deserved it. They asked for feedback for each version and they got pages and pages of comments, suggestions, and bug reports. This in turn resulted to significant improvements to the app.
If you have karma, you have competitiveness, and if we really want to hair-split that in itself can result in scenarios that stop development..
No one wants development to stop, and we both agree that there should be a development karma system of some sort. What I guess we don't agree on is what the karma is for. I look at it as a reward for developers for making good apps and continuously improving them thru end-users' feedback. If you look at it at the 'competitiveness' point of view, well, that is something I didn't think about nor want to go in to.
I agree with Reggie. I don't want to see developers rewarded for competing with each other as much as I want them competing with themselves (ie, continuously taking feedback and putting it into the best work they can do, etc).
In the broader sense, I would prefer developers be rewarded for collaborating with each other. Now-- the challenge is to measure THAT.
A library wouldn't even appear there in the first place. Maybe package maintainers could get an additional x points for each package that depends on one of theirs?
Ooo... that's a start...
Commit count is a very unreliable stat (as people have different (D)VCS usage patterns), and quite often newcomers or non-regular contributors just send patches to core developers, who then apply them to trunk.
Ohloh does analyze more than just numbers of commits. They look at lines of code, comment % etc. The "Contributor Fact" API (https://www.ohloh.net/api/reference/contributor_fact) is quite interesting.
As for non-regular contributors not getting credited, that depends entirely on the culture and tools of the project. Git (that is anyway available on Garage (http://wiki.maemo.org/Git_For_Garage)) makes it quite easy to credit even people just sending simple patches.
I'm not saying this should be the only (or even the main) source of karma from applications, but it would be a nice addition to the formula.
attila77
2009-09-14, 15:53
I agree with Reggie. I don't want to see developers rewarded for competing with each other as much as I want them competing with themselves (ie, continuously taking feedback and putting it into the best work they can do, etc).
That's why I oppose hints like 'top x karma developers/holders will get into the discount program' - that's a direct call for a karma race.
Ohloh does analyze more than just numbers of commits. They look at lines of code, comment % etc. The "Contributor Fact" API is quite interesting.
Yes, I'm familiar with Ohloh metrics (been an Ohloh member myself for some time now), I'm just saying that it's a very rough indicator - lines of code and amount of changes say nothing about the genius contained within that code.
Yes, I'm familiar with Ohloh metrics (been an Ohloh member myself for some time now), I'm just saying that it's a very rough indicator - lines of code and amount of changes say nothing about the genius contained within that code.
Genius contained within the code should probably be apparent from other factors in the karma count - app downloads, ratings and so forth.
Genius contained within the code should probably be apparent from other factors in the karma count - app downloads, ratings and so forth.
Absolutely, which brings us back to arguments that this should (must?) be a mixed metric.
Time to start getting conclusions.
I was actually thinking to send the link of the Brainstorm proposal and this thread to the maemo.org contributors that got a device discount last week. If a consensus doesn't come alone, then propose that someone from the Council would take the action of coming up with a decision during the November sprint.
Traffic on various threads show there are too many corner cases to come up with a solution that is better than the current one. Better in the sense that it provides developers of useful programs with a test device.
Is it possible to allow the community to thumb up on the entries in the Fremantle_Developer_Device_Queue?
You could Brainstorm for that. :)
Time to start getting conclusions.
I was actually thinking to send the link of the Brainstorm proposal and this thread to the maemo.org contributors that got a device discount last week.
I think it's a good idea, at least we can get more opinions/POV.
I think it's a good idea, at least we can get more opinions/POV.
... but then I thought why bothering. :) If someone cares so much about karma and is so unhappy about the current way of handling it, he or she would be here active helping to get a solution. If you need to chase people is because they are not unhappy (one could say).
I have put this in the queue of the maemo.org development process: http://wiki.maemo.org/Maemo.org_proposals#New_.26_Renewed
utteputtes
2009-12-21, 19:41
I am afraid I have to say that I don't trust popularity contests. They are based on the false premise that popularity (what can be measured) truly correlates with quality (what people really want to use).
Couple points:
What happens if application, although quite solution, is still the only mean to accomplish the something? Giving points and encouraging route towards road to nowhere surely can't be the correct thing to do!
Even if there were nearly perfect applications many people would use and thus probably vote (knowingly or nonknowingly) even the worse ones due information asymmetry.
The raw data lacks points for reasoning that could support strategic decision making. If proper marketing study would have to be made anyways why not then start with it and stop messing with the whole argumentum ad populum mess?
Tbh the setting of the original question in this thread and in brainstorm sets dangerous agenda. Liked probably because people instrisicly love being part of decision making process but misleading nevertheless. Experts are still experts. Dig that word "expert" from some better dictionary and ponder about that - I urge you all!
attila77
2009-12-21, 20:19
Let's take a step back and look at the bigger picture. Karma is not an 'excellence in engineering' award, nor is it a voting mechanism. Also, what you are suggesting is even more dangerous - someone 'higher up' classifying apps into 'worse' and 'near perfect' is basically a call to halt all innovation. Let developers scratch their itches the way the want to (within platform recommendations), and let people say what they think of those scratchers (+help improve them). As easy as that.
Karma based on measured software updates is a good metric because it is best measure of.
1) Userbase that actually wants to use the s/w
2) Development activity
Number two could be 'abused' by developers releasing many small updates. But even that is not such a bad thing.
I believe some of the downsides will end up being self-correcting anyway.
There seems to be quite many votes already. Any plans to actually implement this in the near future? :)
There seems to be quite many votes already. Any plans to actually implement this in the near future? :)
We're discussing the proposed changes at the moment: http://n2.nabble.com/Sprint-task-Refine-the-karma-system-td4293910.html#a4293910
vBulletin® v3.8.8, Copyright ©2000-2024, vBulletin Solutions, Inc.