maemo.org - Talk
Page 1 of 3 1 23
Show 40 post(s) from this thread on one page

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Maemo 5 / Fremantle (https://talk.maemo.org/forumdisplay.php?f=40)
-   -   Making Python faster (for fmms initially) (https://talk.maemo.org/showthread.php?t=50737)

lcuk 2010-04-22 10:56
Making Python faster (for fmms initially)
 
Hi,
looking for suggestions/ideas/code/methods for making python faster!

in #maemo irc, x-fade remembered about a pylauncher project, but does not know of its active status in fremantle:
http://pylauncher.garage.maemo.org/

what other things can we do to effectively speed up python
simple things, suggestions metrics etc
does removing the comments make a noticable difference to script runtime?
mini ad-hoc brainstorming session..


================================================== ====
Update, added wiki page results

A couple of weeks later and after much discussion and testing we have a fairly decent set of guidelines and best practices \o/

Performance_Considerations_for_Python_Apps

thanks to epage for the excellent wiki writeup :)

Bundyo 2010-04-22 11:04
Re: Making Python faster (for fmms initially)
 
Is this possible on ARM:

http://psyco.sourceforge.net/

un-named_user 2010-04-22 11:06
Re: Making Python faster (for fmms initially)
 
I'm not a developer so not 100% sure, but this thread is somewhat similar/hopefully helpful.

http://talk.maemo.org/showthread.php?t=47850

Edit: just noticed that the other thread is only for improving the inital startup times and not in general. My bad

lcuk 2010-04-22 11:11
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by un-named_user (Post 622312)
I'm not a developer so not 100% sure, but this thread is somewhat similar/hopefully helpful.

http://talk.maemo.org/showthread.php?t=47850

Edit: just noticed that the other thread is only for improving the inital startup times and not in general. My bad.
don't apologise - its all part of the same thing!

noobmonkey 2010-04-22 11:20
Re: Making Python faster (for fmms initially)
 
Interesting post....... I feel i may be able to provide a few tests to back up issues....

Healthcheck currently has about 300 redundant lines of code (Comments + Blank lines) probably 25% of the total.

I'll run a version of it without all of those and compare load times :D

un-named_user 2010-04-22 11:25
Re: Making Python faster (for fmms initially)
 
@lcuk
My damn N900 ate the smiley for some reason & made me look like a sad panda :p

Anyways good to see the overall scheme includes start up improvements too.

mikec 2010-04-22 11:25
Re: Making Python faster (for fmms initially)
 
The import method of not importing everything has some postive impacts, though probably small.

- reduce startup time
- reduces code size as your code no longer has to refer to the base class. ie qtcore.qrectangle becomes just qrectangle
- not sure if references become a bit faster internally, but feels right to me.

I would also use constants where possible rather than have them values recalculated, as is often the case in code.

given most heavy lifting code is done inside the libraries like qt I,m not sure python is much slower, and if you ae not using libraries why not especially given the richness of them. of course YMMV

lcuk 2010-04-22 11:33
Re: Making Python faster (for fmms initially)
 
from irc:
Quote:
<frals> right fixed the gtk import now anyway so the daemon doesnt import it, not sure about startup time but its taking 13MB less in memory :P
that sounds like an improvement right off the bat!

johnel 2010-04-22 11:33
Re: Making Python faster (for fmms initially)
 
When we talk about making Python "faster" do we mean:
(a) perceptionally faster (e.g. time to reach first screen)
(b) execution speed

attila77 2010-04-22 11:34
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by Bundyo (Post 622310)
Is this possible on ARM:

http://psyco.sourceforge.net/
The concept, yes, but isn't really popular on ARM as it uses quite a bit of memory, which is often not in abundance in the embedded world. There were pioneering attempts on ARM (not the 'real' psyco, that one is i386+ only), but I have no idea what phase did they get to.

Also, for tricks using preloaders: Please consider what that does to memory - keeping GTK and/or QT bindings in there is quite a burden, think it through before you commit to such a thing.

lcuk 2010-04-22 11:37
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by johnel (Post 622355)
When we talk about making Python "faster" do we mean:
(a) perceptionally faster (e.g. time to reach first screen)
(b) execution speed
any/all/both!
its the same discussion at this point! :)

Joorin 2010-04-22 11:48
Re: Making Python faster (for fmms initially)
 
Take it into the measurable realm by doing some benchmarking and profiling. When it comes to programming in general, picking the right algorithm that keeps down the complexity gives several magnitudes more effect than trying to micro optimize in lower layers.

When it comes to the Python run-time, check how it's compiled. What optimization is used? How will it affect what Python does the most (whatever that might be)?

What's the aim? 1% faster? 10%? Depending on the goal, different methods might have to be combined. With a properly compiled Python runtime, good choice of algorithms and a reasonable memory footprint that doesn't cause too many cache misses, 1-5% might very well be possible.

And don't forget the Linux environment. What scheduling is used in the kernels delivered by Nokia and the community? Has anyone looked into backporting the new ones, like BrainF u c k scheduling, that's specifically aimed at making the desktop faster? This might make the N900 as a whole faster.

So, lots of variables.

attila77 2010-04-22 11:51
Re: Making Python faster (for fmms initially)
 
Far from a ready solution, but worth keeping an eye on (smells like Harmattan candidate material):

http://code.google.com/p/unladen-swa...ki/ProjectPlan

TheBootroo 2010-04-22 12:09
Re: Making Python faster (for fmms initially)
 
use C++/Qt4 instead of PyQt

johnel 2010-04-22 12:22
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by TheBootroo (Post 622407)
use C++/Qt4 instead of PyQt
Verrrrrrry funny.

That is as helpful as a chocolate matchstick!

johnel 2010-04-22 12:26
Re: Making Python faster (for fmms initially)
 
What we need are some benchmarks of how things are just now.

See where the bottlenecks are. Then start tweaking things in a controlled manner and compare the bechmarks to the original result.

If we are talking about a specific program then we need to find what routines are the most intensive then maybe re-implement the routine in C/C++ an link it into the Pthon-based program.

Profiling is probably the first step.

Python profiling stuff

Helmuth 2010-04-22 12:56
Re: Making Python faster (for fmms initially)
 
Okay, not really a speed improvement. But the apple way to do the trick:

http://lists.maemo.org/pipermail/mae...er/021478.html

:)

Khertan 2010-04-22 19:05
Re: Making Python faster (for fmms initially)
 
Nouvelle note 47

Quote:
Originally Posted by TheBootroo (Post 622407)
use C++/Qt4 instead of PyQt
I'm askin for a ban for a such stupid comment !

Please consider that a laucher use memory and in some case can be slower.

Anyway ... there is simple things to optimize python scripts :

- When available, prefer module methode instead of self made in python, as most of module are in c. So really faster
- Avoid when possible too much search in namespace and prefer local method variables it s faster to assign a methode from namespace in a local var when it s called in a loop with more than 10 iterations
- abuse of profile to see where your code is leaking time (available on device)
- abuse of pylint
- instead of doing many test case, use exception : a simple example :
Code:
  if (mydiv#0):
    res=res/mydiv
  else:
    res=0
use:
Code:
  try:
    res=res/mydiv
  except NullDivisionError,e:
    res=0
- memorize pattern can help
- avoid cos , sin and tan in a game , use a precalculate array (for vectormine it s optimize fps from 3fps to 60fps)
- and again abuse of profile
- avoid converting data type in loop... Example QString to str
- use existing module when available : PyGtKEditor do all his parsing in python ... it s can be slow on large file, khteditor do the things clearly faster by using some Qt native class.

An other things to doesn't freeze the ui and to not slow down the device by using intensive thread is to divide tasks in very small tasks and to use a sigleton with gidle :
Code:
import gobject

class _DeferClass(object):
  _calls=[]
  _ref=None
  def __new__(cls,*args,**kw):
    if cls._ref is None:
      cls._ref = super(_DeferClass,cls).__new__(cls,*args,
                                                    **kw)
    return cls._ref

  def __len__(self):
    return len(self._calls)

  def __call__(self,func,*args):
    def NextCall():
      (func,args)=self._calls[0]
      func(*args)
      self._calls=self._calls[1:]
      return self._calls!=[]
    if not self._calls: gobject.idle_add(NextCall)
    self._calls.append((func,args))
I ve many example to give and there is so man things to say that i ll be able to write a book about py opimization.

So if you need so help optimizing code i can maybe help you.

mikec 2010-04-22 19:15
Re: Making Python faster (for fmms initially)
 
Khertan you are Sweenie Todd :D

I'm eating Py, that s**t is so advanced I dont even know what it does:D

epage 2010-04-22 19:19
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by lcuk (Post 622298)
does removing the comments make a noticable difference to script runtime?
I can't say off of a benchmark but comments affect parsing. If pyc (or even better?) pyo files are already generated, stored, and run then comments shouldn't matter.

GCDialer (old name for Dialcentral) use to remove extra whitespace and merge multiple python files into one. I didn't feel the cost/benefit was worth it but didn't benchmark it.

I did do some benchmarking and found that class assignments are slow. When creating a lot of class variables (like pre-compiling a bunch of regexes) it was faster to create them on the instance. I normally have only one instance so no extra copies to worry about.

For general python optimizations there is already a decent guide
http://wiki.python.org/moin/PythonSpeed/PerformanceTips

Quote:
Originally Posted by attila77 (Post 622382)
Far from a ready solution, but worth keeping an eye on (smells like Harmattan candidate material):

http://code.google.com/p/unladen-swa...ki/ProjectPlan
Benchmarks are so-so for now but they haven't gotten a chance yet to do the optimizations they wanted. The real killer is this is very memory intensive which the acceptance into CPython is conditioned on improving memory usage. Start time is also an issue for now.

http://www.python.org/dev/peps/pep-3146/

Long term, PyPy is the project I'm more excited about. They've recently added support for CPython extension modules which was a major blocker for transitioning. Speed is a mixed bag. I'm unsure about start time or memory usage though.

http://speed.pypy.org/overview/

epage 2010-04-22 19:23
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by Khertan (Post 623199)
An other things to doesn't freeze the ui and to not slow down the device by using intensive thread is to divide tasks in very small tasks and to use a sigleton with gidle :

import gobject

class _DeferClass(object):
_calls=[]
_ref=None
def __new__(cls,*args,**kw):
if cls._ref is None:
cls._ref = super(_DeferClass,cls).__new__(cls,*args,
**kw)
return cls._ref

def __len__(self):
return len(self._calls)

def __call__(self,func,*args):
def NextCall():
(func,args)=self._calls[0]
func(*args)
self._calls=self._calls[1:]
return self._calls!=[]
if not self._calls: gobject.idle_add(NextCall)
self._calls.append((func,args))

I ve many example to give and there is so man things to say that i ll be able to write a book about py opimization.

So if you need so help optimizing code i can maybe help you.
An alternative is to use a worker thread per set of logic data to avoid locks and then wrap it in classes to handle the jumping between the main loop and my thread.

AsyncLinearExecutor code:
https://garage.maemo.org/plugins/ggi...51e98721401e72

Code that uses AsyncLinearExecutor:
https://garage.maemo.org/plugins/ggi...51e98721401e72

Khertan 2010-04-22 21:14
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by epage (Post 623233)
An alternative is to use a worker thread per set of logic data to avoid locks and then wrap it in classes to handle the jumping between the main loop and my thread.

AsyncLinearExecutor code:
https://garage.maemo.org/plugins/ggi...51e98721401e72

Code that uses AsyncLinearExecutor:
https://garage.maemo.org/plugins/ggi...51e98721401e72
Maybe but mine work also if the app is in background and we didn t want eat too much cpu of the app in front

epage 2010-04-22 21:25
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by Khertan (Post 623392)
Maybe but mine work also if the app is in background and we didn t want eat too much cpu of the app in front
I guess I'm not seeing the case but oh well, both work with each probably best for different needs

Khertan 2010-04-23 05:42
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by epage (Post 623413)
I guess I'm not seeing the case but oh well, both work with each probably best for different needs
Yep both both are different solutions for different use case.

dbrodie 2010-04-23 11:40
Re: Making Python faster (for fmms initially)
 
Well, depending on the case, Cython can be a very handy tool...

Khertan 2010-04-23 12:31
Re: Making Python faster (for fmms initially)
 
Also :

- Avoid while loop when you can use a "for foo in foo_list"
- use xrange instead of range (removed in python 3 so it s not really a good idea for compatibility)

ah and of course, Read this : http://wiki.python.org/moin/PythonSpeed/PerformanceTips

lcuk 2010-04-24 01:49
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by noobmonkey (Post 622330)
Interesting post....... I feel i may be able to provide a few tests to back up issues....

Healthcheck currently has about 300 redundant lines of code (Comments + Blank lines) probably 25% of the total.

I'll run a version of it without all of those and compare load times :D
hey Greg, did you manage to try this by chance?
it propbably won't make too much difference but if we can get a line count before and one after if there is any kind of difference it can go into the melting pot :)

noobmonkey 2010-04-25 19:22
Re: Making Python faster (for fmms initially)
 
Ahaaaaaaaaaaa i'm back! and had time to do it! yay!

First test - normal code
2104 lines of code
580 blank lines
215 code lines
Load time from icon click to fully loaded - 10.04 seconds

Second Test - Cleared up code
2104 lines of code
0 blank lines
80 code lines
Load time from icon click to fully loaded - 9.25 seconds

Ok, it was a basic test, but i ran them side by side for a while with nothing else loaded... the load times where almost exactly the same each time. So... yes blank lines do affect load up!

My next test, later this week will be to remove redundant lines into sensible functions - should cut my lines of code down a lot...... so will compare that against these results.

:D

quick edit :-
Third - Cleared up code!!
1469 lines of code
0 blank lines
80 code lines
Load time from icon click to fully loaded - 8.40 (5 tests , from 8.09 to 8.60)

(Sorry code lines probably don't mean much - i meant comment lines!)

lcuk 2010-04-25 19:35
Re: Making Python faster (for fmms initially)
 
noobmonk3y - so you just saved 20% of the load time by scrubbing your code?

thats remarkable (tho extreme, we like comments)

i did not expect the difference to be so great.
if we could strip them from debs as part of packaging for most python apps that would be wicked
(keep them in source of course)

attila77 2010-04-25 21:39
Re: Making Python faster (for fmms initially)
 
@noobmonkey: use the import trick from my presentation, that way you can save the time required for parsing (empty lines or not).

Jaffa 2010-04-25 21:56
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by epage (Post 623221)
I can't say off of a benchmark but comments affect parsing. If pyc (or even better?) pyo files are already generated, stored, and run then comments shouldn't matter.
As part of Hermes' Makefile I have:

Code:
compile:
    py_compilefiles src/*.py
This should pre-compile "Python .py source files into .pyc or .pyo bytecode format."

I haven't done any performance testing, but if packagers don't do this, and the Python app is being run as user (and it's installed in /opt or /usr), Python will have to do parsing every time: it won't be able to write the bytecode back to disk for next time, as "user" usually won't have permission to the install directory.

attila77 2010-04-26 07:17
Re: Making Python faster (for fmms initially)
 
An alternative for handling this ^ is in the postinst/postrm script on the device, that way the debs are kept smaller and apt needs to track less files (at the cost of doing that yourself and a longer install time). Plus, should a system update be shipped with a pyc incompatible Python version, reinstalling the packages updates the pyc/pyo-s without any maintainer intervention.

X-Fade 2010-04-26 11:38
Re: Making Python faster (for fmms initially)
 
Precompiling should save quite a lot on parsing as it needs to simple read less bytes from storage and doesn't have to do the parse part.

Joorin 2010-04-26 11:51
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by X-Fade (Post 628190)
Precompiling should save quite a lot on parsing as it needs to simple read less bytes from storage and doesn't have to do the parse part.
Precompiled code in an evaluated environment speeds up loading a great deal.

But remember that the time it took to start the program always gets spread out over how long time you actually use the program. 10s to start for an application that you use 2s (for example getting temperature data from your favourite source and displaying it) is painful. 10s to start for an application that you use for 20m is something quite different.

But sure, it's always nice when applications start almost before you knew you wanted to use them. :)

dalonso 2010-04-26 15:55
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by attila77 (Post 627924)
An alternative for handling this ^ is in the postinst/postrm script on the device, that way the debs are kept smaller and apt needs to track less files (at the cost of doing that yourself and a longer install time). Plus, should a system update be shipped with a pyc incompatible Python version, reinstalling the packages updates the pyc/pyo-s without any maintainer intervention.
This matches what's in the Debian Python Policy, as well:

http://www.debian.org/doc/packaging-..._packages.html

2.6 Modules Byte-Compilation

If a binary package provides any binary-independent modules (foo.py files), the corresponding byte-compiled modules (foo.pyc files) and optimized modules (foo.pyo files) must not ship in the package. Instead, they should be generated in the package's postinst, and removed in the package's prerm. The package's prerm has to make sure that both foo.pyc and foo.pyo are removed.

A binary package should only byte-compile the files which belong to the package.

The file /etc/python/debian_config allows configuration how modules should be byte-compiled. The postinst scripts should respect these settings.

Pure Python modules in private installation directories that are byte-compiled with the default Python version must be forcefully byte-compiled again when the default Python version changes. Public Python extensions should be bin-NMUed. Private Python extensions should be subject to binary NMUs every time the default interpreter changes, unless the extension is updated through a .rtupdate script.

epage 2010-04-26 16:01
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by lcuk (Post 627333)
noobmonk3y - so you just saved 20% of the load time by scrubbing your code?

thats remarkable (tho extreme, we like comments)

i did not expect the difference to be so great.
if we could strip them from debs as part of packaging for most python apps that would be wicked
(keep them in source of course)
A problem with scrubbing files is when I get a log from my users with a traceback, I have to do extra hunting to figure out what code caused the exception.

Prioritize this nugget wherever you want.

Texrat 2010-04-26 16:08
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by lcuk (Post 627333)
thats remarkable (tho extreme, we like comments)
I agree, although with good formatting and use of descriptive variable names, code can be highly self-documenting.

And I wonder if use of in-line comments versus separate-line comments makes a difference? Greg, can you test that also?

zimon 2010-04-26 16:25
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by dalonso (Post 628464)
Public Python extensions should be bin-NMUed. Private Python extensions should be subject to binary NMUs every time the default interpreter changes, unless the extension is updated through a .rtupdate script.
What is NMU?

zimon 2010-04-26 16:33
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by epage (Post 628476)
A problem with scrubbing files is when I get a log from my users with a traceback, I have to do extra hunting to figure out what code caused the exception.
And you loose one of the benefits of Python; readability.
I find it not a good solution to strip comments and remove empty lines for speed sacrificing readability, therefore increasing the probability of bugs and making debugging harder.

attila77 2010-04-26 16:48
Re: Making Python faster (for fmms initially)
 
Quote:
Originally Posted by zimon (Post 628509)
What is NMU?
Non-Maintainer Upload (it's a Debian thing, not Python - don't freak out :) ). IIRC Happens when the maintainer goes AWOL or a system change/security issue/bug report required response from the distro folks.

http://wiki.debian.org/NonMaintainer...w&redirect=NMU


All times are GMT. The time now is 10:31.
Page 1 of 3 1 23
Show 40 post(s) from this thread on one page

vBulletin® Version 3.8.8