Python Tip of the [day, week, month]

PTOTD

Starting tomorrow, I'll post a Python tip on a regular basis. I cant promise a PTOTD, but it'll be more often than once a month, so that identifies the boundaries.

Ok, I lied, I'll start with one right now:

Tip #1

python -i script.py

What it does: At the conclusion of the execution of script.py, instead of exiting, the python interpreter stays in interactive mode, with everything ready to be printed or debugged.

François
@f_dion

Project Night: A red Raspberry Pi, 126 LEDs and a job

Project night

Every month, through PYPTUG, we set aside 2 nights for the Pythonistas and Pythonistas to be of the Piedmont Triad community. Tomorrow (wednesday the 20th of November) is one such night.

So what do we do there? The meetup page says:

"Do you have a project you want to show off? Or do you need a second set of eyes on your code? Would you like to hack on an open source project? Perhaps you need help getting a module on a specific platform? Don't have a project but would like to help others or to learn more Python?
Whatever the case may be, project nights are your opportunity to get some help or to help others, so bring your laptop (or Raspberry Pi or whatever device that can run Python) and let's get things going."

Red Raspberry Pi

So, I always show up at these project nights with a few potential projects. For example, hacking on Brython, or writing a good music collection manager in Python, or playing with a red Raspberry Pi. Red? Yes, this kind:

 

Pi Lite

But that wouldn't be a project in itself. I'll bring over a Pi Lite. What is that?


126 ways to have fun.

Spectrum analyzer? network traffic display? game of life? cool scroller? Who knows what i'll end up doing, but it'll be fun, I'm sure.

A job?

Oh yes, almost forgot. So this project night is in Winston Salem:

Inmar 2650 Pilgrim Court, Winston Salem, nc

And it just happen to be that there are some openings for Python developpers, doing web services and focusing on Python. If that sounds like fun, get in touch.

Winston Salem is a great place to live and work (and that's me saying this, coming from cosmopolitan Montreal). Plus Inmar is migrating downtown pretty soon, in the Wake Forest Innovation Quarter. So come on, take the first step and possibly come and join me and other enthusiastic Pythonistas at Inmar and drop me a note, or even better, email me your resume.

Requirements:
Experience with Python (web services a plus) and with Postgresql (or even mysql).

François
@f_dion

toolz and curry

Lightning talk

Doing a quick lightning talk tonight for PYPTUG @WFU and it wont be about spanners and food as the title might imply, but about...

Functional programming

Yes, about functional programming in an imperative by design language (Python). And it's a lightning talk, so it'll be very superficial. But hopefully interesting nonetheless.

My first experience with pure functional programming was in the 80s with the Miranda programming language. If one counts impure functional such as LISP, scheme etc, then that would be Logo as my first.

Back to Python

One of the most appealing Pythonic technique I use on a regular basis, is functional. But more on that later. Here in this talk, I'll focus on a new module called Toolz (the result of a merge of functoolz and itertoolz - with a z, not the standard library functools and itertools).


François
@f_dion

IEEE 754/854, when it is needed

Linkedin

There was a post on linkedin IEEE Computer Society group, lamenting the division by zero exception raised by Python, instead of returning infinity (or -infinity). I replied, but it is not that readable. So I'm putting the properly formatted reply on my blog. [edit: no comments followed on linkedin, but many appeared on the original source after that]

IEEE decimal

If you want IEEE 754/854 decimal behaviour, what you want to use is the decimal module:

>>> import decimal
>>> help(decimal)

The above typed in the python interpreter will provide all the help you need. The examples are a little verbose, however.

ExtendedContext

If you want to be shorthand, you can copy/paste the following at the top of your program:

from decimal import setcontext,ExtendedContext, Decimal as ieee
setcontext(ExtendedContext)


Then whenever you want to use ieee behaviour use ieee(value):

>>> 1/ieee(0)
Infinity
>>> -1/ieee(0)
-Infinity

Python does it right

It is good that it has to be explicitly stated that you want this. A division by zero should raise an exception when dealing with financial data[*]. I would recommend numpy if dealing with scientific data.


François
@f_dion

* Unless you are calculating my bonus...

$1 in Python

Interesting question

"Why cant I use $1 in a Python script to get the first argument to a script? What is the easiest way to do something like that in Python?"

$1 in a shell script gives us the first argument to the script, BTW.

I get a lot of questions through email, and in person. This one was part of an interesting exchange. The reasoning went like this, since the shell is processing the script, I should be able to use $1.

The problem is that it doesn't work like that. Dennis Ritchie introduced the #! (pound bang or shebang line) concept at Bell Labs and announced it publicly in 1980. In the exece function of sys1.c could be found the following:

 else if (u.u_exdata.S[0]=='#' && u.u_exdata.S[1]=='!' && indir==0) {
  cp = &u.u_exdata.S[2];
  while (*cp==' ' && cp<&u.u_exdata.S[SHSIZ])
   cp++;
  u.u_dirp = cp;
  while (cp < &u.u_exdata.S[SHSIZ-1]  && *cp != '\n')
   cp++;
  *cp = '\0';
  indir++;
  iput(ip);
  ip = namei(schar, 0);
  if (ip==NULL)
   return;
  goto again;

So, once it was established that a script started with #! and a variable number of spaces, what followed was the path to an executable that would process this script. No search, so it has to be a full path.

In that regard, if the line starts with #!/usr/bin/python or #!/usr/bin/env python, the shell is out of the loop and no $1. Python will process the script. There is an sh module that makes it appear like we are mixing Python and shell in one script, but it is still all Python code.

Simplest shell script

#!/usr/bin/bash

echo $1

Simplest Python

#!/usr/bin/env python
from sys import argv as s


print(s[1])

Fairly similar except we had to specify an import statement, and I aliased argv to s so it looked like $ in the shell script.

The "problem" (or difference) with the Python script is that it will error out if no arguments are given when run:

$ ./arg.py
Traceback (most recent call last):
  File "./arg.py", line 5, in <module>
    print(s[1])
IndexError: list index out of range

A closer equivalent

#!/usr/bin/env python
from sys import argv as s

print(s[1] if len(s)>1 else '')

Here we are simply testing to see how many items are in the argv list. The first item is the script name, so if we have >1 items, we have at a minimum one argument, so it is safe to get [1]. If not, we will return an empty string, just like a shell script would do. A try / except IndexError would have also worked in this context.

See also

I would also investigate argparse, opster and docopt. argv is really bare metal and there is no point in reinventing the wheel.


François
@f_dion

The Raspberry Pi @ForsythTechCC IEEE club

Last week the ForsythTech IEEE club got a Raspberry Pi. Glad to see it being put to good use, Carroll Perkins just sent me the following:

"This Thursday, Sept 26th, at 3PM in Hauser 332, the ForsythTech IEEE Students meeting is a demo by Bobby Garriott of the Raspberry Pi computer hook up.  LCD screen, keyboard and mouse hook ups will be demonstrated.

The Raspberry Pi is a $35 computer board.  Ideal for embedded applications of all types including robotics, CNC and 3D printer applications. "

RPi MOOC en espanol

Donde

http://telescopio.galileo.edu/curso/raspberry-pi/

Cuando

El 14 de octubre 2013.

Información

Indican en la pagina que van a abarcar todo eso:

• Instalación, Configuración, Accesorios y demás aspectos a considerar
• Demos de instalación Wheezy y otras distribuciones
• Introducción a Python
• Introducción a la programación en Python para Raspberry Pi
• Ejemplo completo Python
• Modulo GPIO del Raspberry Pi para conexiones externas
• Elementos básicos de hardware y como controlarlos con el GPIO
• Próximos Pasos: Proyectos y Comunidad

Me parece muy genial. Es algo similar a los taller PyHack que hacemos en Carolina del Norte, pero en la red.

François
@f_dion

The lost art of digital entomology

Slides

I had forgotten to push the slides and code to bitbucket. This is from my PYPTUG talk from june. You can get the files here:

https://bitbucket.org/fdion/pyptug/src/a64e0c283150/debug?at=default


François
@f_dion

@surgeterrix to XOR or not to XOR

By way of twitter

Taking 2 strings of hexadecimal characters, converting hex to binary values, then xorsum the 2 values, and convert sum back to hex.

The other XOR

So the normal xor will not work on strings, so what do you do?

from binascii import hexlify, unhexlify  
from Crypto.Cipher import XOR  
  
# you can encounter hex strings with no spaces  
encrypted = '556e6e216c606f78217264627364757216'  
# or hex strings with spaces  
key = '01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01'  
  
#let's get binary representations  
b_encrypted = unhexlify(encrypted)  
b_key = unhexlify(key.replace(' ','')) # we remove the spaces  
 
#to see a binary string output, use repr: print(repr(b_key))  
  
cosmo = XOR.new(b_key)  
bishop = cosmo.encrypt(b_encrypted) # yeah, encrypt to decrypt  
  
print hexlify(bishop)  
# the above is what was asked  
# the plain ascii decrypted message is print(bishop)  

I think the (not really) hidden references are pretty obvious...However, even in the decrypted message, there is still a question left... :)

François
@f_dion

A FISH named Blackbox

Boat

Saw an Oracle TV spot during coverage of the America's cup. They emphasize hardware and software working together, data gathering and even some wearable computing. So, that's what my TV spot would be if I had their budget?

FISH

Also known as Fully Integrated Software and Hardware. So, for the past few years I've been working a tight integration of software, hardware and wearable computing. For metrics, data logging, and stuff like that. And it is finally ready to market. I've never posted about my company and my work on my blog before, but I had to share this.

The Blackbox

This latest revision of the Blackbox is running Linux, and all the major software has been written in Python. In its most basic form, it can be used as a digital signage, and yet can be used to do really cool stuff, like proximity sensing and machine area networking. Walk up to a piece of equipment, any piece of equipment, and you get to see realtime metrics on your tablet, for that machine. And wearable computing.

Unveiling the FISH

So I will be unveiling this at the Triad business innovation expo in Winston Salem, NC




Date/Time
Date(s) - Thursday, September 12, 2013
8:00 am - 3:00 pm
Location
Benton Convention Center
301 W. 5th Street
Winston-Salem, NC

So if you want to check it out, I'll be at Dion Research's booth (431) demoing the FISH solution. And did I say wearable computing, just like in the TV spot? I cant say more than this, I'll do live tweeting and unveiling on Thursday.

François
@f_dion

From Java to Burma

Hands on Python

This month at our local Python user group (PYPTUG), I'll do a hands on session. Could have been in a workshop, code dojo or project night, but it'll be part of the normal monthly meeting.

Attributes, Properties and Descriptors

A lot of people learned C++ or Java in school and some of the normal patterns for these languages are regularly seen in the wild in languages such as C# and Python. Having coded in all of these, and many more, I appreciate greatly some of the features of a given programming language. And when it comes to Python (as Roger Sessions said, paraphrasing Einstein), that "everything should be as simple as it can be, but no simpler".

 The hands on will focus on attributes, and how to keep it simple. And of course, how to get out of trouble, because you have to do something not so simple later, using properties and descriptors.

I will be posting to my bitbucket pyptug repository some material related to the talk, including a log of the interactive session we will be doing.

Audience will be ranging from people who have never coded, to people who have programmed in other languages, to python experts. I love a challenge.

François
@f_dion

Nouvelle version de Brython: 20130817-141536

Téléchargement de Brython

Une nouvelle version de Brython, Brython-20130817-141536, est maintenant disponible sur la page des téléchargement
 

Changements

Depuis la version précédente :

- import amélioré , on peux utiliser les packages : "import X.Y", "from X.Y import Z [as A]", "from . import X" (corrige le bogue 123)
- le protocole des itérateurs utilise maintenant les méthodes __iter__() et __next__() pour tous les types
- "for" les utilisent aussi
- implémentation des compréhensions améliorée : utilise les dictionnaires global et local (corrige le bogue 129)
- améliorations a la console interactive : http://brython.info/tests/interactive_mode.html
- ajout du type "bytes" (incomplet)
- ajout de balises HTML5 et événements DOM manquants
- ajouts a la documentation en français, ajout d'une section pour les modules spécifiques a Brython ; autres parties traduites
- Bouton sur la page de tests pour exécuter tout les tests et retourner les erreurs
- amélioration du module markdown : les _ et * génèrent du HTML différent, tout comme __ et **
- le mot clé del : utilise la méthode __del__() si elle existe

Bogues

• - Corrections :

. #78 : map() and filter() return iterator objects, not lists
. #124 : support of CSS attribute "float"
. #125 : add methods copy() and clear() to lists
. #126 : escaped quotes inside strings
. #127 : DOMNode instances comparison methods (__eq__ and __ne__)
. #128 : class inheritance
. conversion des JS objects dans json.js et $JS2Py
. problème avec les itérations sur instances DOMNode

Autres

• réorganisation : déplacement de JSObject du fichier py_dom.js au fichier py_utils.js
• galerie : ajout d'un jeu de solitaire qui utilise SVG et le glisser - déposer

Vos retours

L’équipe Brython espère vos retours sur cette version, autant pour nous laisser savoir si vous trouvez  des bogues mais aussi au niveau de votre expérience avec Brython, en joignant et participant aux forums Brython.

De plus nous aimerions entendre parler de vos projets (actuels ou futurs) avec Brython sur appareils mobiles (tablettes, etc) et informatique embarquée (Raspberry Pi, Beaglebone black, Cubieboard, etc), surtout au niveau des écrans tactiles, car nous y travaillons en ce moment. Alors, n’hésitez pas a joindre le forum ou a me contacter directement sur mon blog ou sur twitter.

François
@f_dion

200000th visitor

Raspberrypi and Python

In February 2013, this blog reached the 100,000th visitor milestone. Tonight, it just past the 200,000th.

This little experiment has been ongoing for almost a year now. As I had mentioned before, I didn't expect a lot of visitors due to the specificity of the blog, but it was something I had to do. Most projects I do for business, I cant talk about, so I've made up some projects specifically for this blog.

The Raspberry Pi has continued on the road to popularity, and this has exposed many people to embedded computing, to software development and electronics, and that is a good thing. Some other subjects that have been popular over the year have been Brython, ZFS and dtrace.

I've seen people I've introduced to the Pi, to electronics or to replicators (ie. CNC, 3D printers, laser cutters), to Python or Brython come up with some really cool stuff in the past year, and that is ultimately quite rewarding.

The visitor 

So, where is this 200,000th visitor from? Surprisingly from the town of Sanford, NC. I say surprisingly because it is only an hour and a half away from Winston Salem by car.

Sanford, NC

François
@f_dion

Brython 20130817-141536 published

Download Brython

A new version of Brython, Brython-20130817-141536, has just been published on the downloads page

Changes

Here are the main changes since the previous version :

- improved implementation of import, now supports packages : "import X.Y", "from X.Y import Z [as A]", "from . import X" (fixes issue 123)
- the iterator protocol now uses the standard methods __iter__() and __next__() on all types, including built-in types
- the "for" loop implementation has been changed to use these methods
- improved implementation of comprehensions : now uses global and local dictionaries (fixes issue 129)
- improved interactive console : http://brython.info/tests/interactive_mode.html
- partial support of built-in type "bytes"
- add missing HTML5 tags and DOM events
- change documentation in French, add a section for Brython-specific modules ; updated translation of documentation
- add a button on the Tests page to run all the tests in a row and report errors
- improvement to the markdown module : tags _ and * generate different HTML tags, as well as __ and **
- change implementation of the del keyword : now uses the method __del__() if it is defined

Bugs

• - bug fixes :

. issue #78 : map() and filter() return iterator objects, not lists
. issue #124 : support of CSS attribute "float"
. issue #125 : add methods copy() and clear() to lists
. issue #126 : escaped quotes inside strings
. issue #127 : DOMNode instances comparison methods (__eq__ and __ne__)
. issue #128 : class inheritance
. bugs in conversion of JS objects in json.js and $JS2Py
. bug with iteration on DOMNode instances

Other

• package organisation : move JSObject from py_dom.js to py_utils.js
• gallery : add a demo of a solitaire game using SVG drag and drop

Feedback

As always, the Brython team looks forward to your feedback, particularly in terms of bug reports and also in sharing how you are using Brython, by joining and participating in the Brython forum.

We are quite interested in your experience on mobile devices (tablets etc) and embedded devices (Raspberry Pi, Beaglebone black, Cubieboard etc), particularly in conjunction with touchscreens, as we are about to bring further enhancements to Brython. So don't be shy and join the forum. Or at least contact me through my blog or on twitter.

François
@f_dion

A python interactive console in my blog

Brython modes

I had prepped a presentation on Brython for a Python conference and didn't want that to go to waste, so now, dear reader, you get to see all kinds of exclusive material I had made for it. (Don't worry, I'll have plenty more material for PyCarolinas 2013).

The difference is that this is a blog, not a talk, so it will be a good bit more non linear. I'm starting tonight with something very hands on, one of Brython's mode of deployment.

The Brython interactive mode

The brython.js script itself is not included in this page, so it is a hosted interactive mode. We will simply include an iframe to load an interactive console directly from brython.info. This is still quite experimental, as it is found under the tests/ section. It is now using most of my iframe box (was fixed some days back). But it does work pretty well already.

This is basically a zero install, just add the following in your web page and you'll get a brython interactive session (Edited 3-26-2014 to point to new location):  

<iframe src="http://www.brython.info/console.html" width="98%" height="400">Sorry... your browser doesn't support iframe. Time to upgrade or go to <a href="http://www.brython.info/console.html">http://www.brython.info/console.html</a> in a separate tab.</iframe>







So you should see a console above (wont work trough news aggregators). At the >>> | prompt, type:

x = input("yo")

Type the word me in the prompt dialog that will open. Make sure you type exactly that. So what do we have in x? Type the following:

print(x)

me - Ok, that's all nice, but not very exciting. So it does behave like a console. What else can we do?

import webbrowser

So now we have loaded the module webbrowser from the brython standard library (we will come back to that in a future article). Yeah, I know, mind bending, since we are client side. Let's use the module. Type the following:

webbrowser.open("http://en.wikipedia.com/wiki/internet_" + x*2)

Nice! All your base are belong to us, obviously. So, it does take a minute to get the brain wrapped around this concept, but once you do, the world's your oyster. You just have to:

doc <= "think"

:)

Alright, enough geek puns, but it is a friday night afterall. So go and check it all out at http://brython.info . There is documentation in english, french, spanish and portuguese. Feel free to contribute your own translations in markdown format through bitbucket.

If you've been following my blog, you already know about the <= (left arrow) operator. If not, check it out here: http://raspberry-python.blogspot.com/2012/12/brython-browser-python.html

In the early days, brython didn't have a print() keyword. So I had cooked up a quick webprint() that used the <= operator. Of course, 8 months later, we no longer need webprint(). Print works like it should, with stdout. For the interactive mode, it is redirected. Look at the (python) code to see how you can do that by right clicking on the iframe and doing a view frame source (hint, lines 48 to 51).


François
@f_dion

Site brython.info em portugues

Site

Algo novo hoje:

em portugues!

Documentação

O site juntamente com a documentação foi traduzido. Tudo graças aos esforços de Yuri Teixeira.

Grupo

Existe tambem um grupo googlegroups em portugues:

https://groups.google.com/forum/?fromgroups=#!forum/brython-pt

François
@f_dion
 

Of videogames, arduinos and 10 year olds

Vpython

That's Mr. Visual Python to you (the visual module, that is). At the last PYPTUG meeting, Aaron Titus gave a talk on Visual Python. I think it was about writing (well, mostly playing) videogames under the veneer of learning physics. :)

All joking aside, it was a really great talk. It was also the first time we were using Google+ Hangout to help some members to check the presentation. And particularly a father and daughter who would not have been able to make it in time for the meeting. More on that further down.

Pyfirmata lightning talk

So I gave a lightning talk on Pyfirmata and Arduinos. It was part of my talk at PyCarolinas last year but didn't have time to go into the code detail (plus another talk earlier that day had covered really extensively serial communications). And it's really about serial communications (be it USB, bluetooth or plain old RS-232/rs-422) and microcontrollers (it's not just for the 'duinos). I ran the python code on a Raspberry Pi, but you could use pretty much any computer that can run python.

The protocol is firmata: http://firmata.org

The python module is Pyfirmata: https://github.com/tino/pyFirmata

To install, you can use :
pip install pyfirmata
or
easy_install pyfirmata

Since both pip and easy_install automatically install dependencies, that's all there is on the host computer side.

Or from source using setup.py, but you will have to install the serial module.

You dont have to handle the serial stuff yourself, just point to the right /dev/ (serial, usb or bluetooth), start an interator and start using read(). Pyfirmata handles the serial itself and sending/receiving the sysex bytes (Firmata is a MIDI inspired protocol).

To install the firmata firmware on the Arduino, start the IDE and go to:



The bitbucket repository for my talk (presentation and simple example that reads values from analog port 0 on the arduino) is at:
https://bitbucket.org/fdion/pyptug

 

The young coder

"After watching the stream, this morning I left my 10 year old building spheres and boxes in VIdle.  She'd never done programming of any kind before last night." -- Ryan

I think this speaks volume about Python and the visual module. And that's what PYPTUG is all about. Now we just got to get her a Raspberry Pi.

François
@f_dion

Trip to the bookstore

Community 

Part of a good local technical community is having local technical schools, hackerspaces, computer stores, electronics parts distributors, libraries and book stores with good technical books, and of course user groups.

For PYPTUG community news, I try to always make a note of Python related things wherever I go so I can share it with fellow community members in the Piedmont Triad.

This is suspiciously easy

Python is easy to get into. In fact, here's something on the back of a book I saw at the local Barnes And Nobles (Hanes Mall, Winston Salem, NC):

Hello! Python

Couldn't help but to share that with my wife. It is on the back of the book Hello! Python by Anthony Briggs.

And that too

Using the Raspberry Pi in various projects is also quite easy, and having tutorials or a guide makes it that much easier. Of course I have plenty of tutorials on my blog, but if you are looking for a Raspberry Pi book you can hold in your hands, there are a few available now, and if you are in the Winston Salem area, you can pick these up without having to wait for the postman:

Raspberry Pi User Guide by Eben Upton

Getting Started with Raspberry Pi

Programming the Raspberry Pi

Nice to see this on the bookshelves.

Adding Arduino to the mix

And speaking of Raspberry Pi, tomorrow I'll be doing a lightning talk on something I had talked about back in 2012 at PyCarolinas. I was going to get into more details at PyOhio, but as you've no doubt noticed, I'm not on the speaker list, so instead I'm covering this locally at PYPTUG.

So what is it? A super easy way to interface with microcontrollers, including the Arduino, using Python. Did I say super easy? Suspiciously easy :)

François
@f_dion

Pidora and Python

Hands On: Raspbian alternative on Raspberry Pi Part Deux

Last month, I looked at FreeBSD and Arch Linux and demoed these in a hands on session. This month, one of the Pi I'll have for the hands on this coming monday at WFU will be running Pidora.

If you are used to Raspbian for your Python coding, you'll find that there are a few differences, such as using yum instead of apt-get. Development packages end in -devel instead of -dev. All of that is quite normal since pidora is really Fedora and Raspbian is debian.

What I've found though is that if you are not willing to build quite a few packages yourself, it is still a little early to adopt pidora. I've not been able to get a satisfactory (ie, full library support) version of pygame going (no package available in the pidora repository at this time). No mpeg, no framebuffer support etc.

On the plus side, it boots very quickly (faster than arch or raspbian) and the shutdown is also extremely fast. I suspect that within a few months, the repository will be a bit more complete and will make python development easier. I'll revisit the OS later this year.

François
@f_dion

Raspberry Pi, the other OSes

Hands On: Raspbian alternative on Raspberry Pi

At PYPTUG, for the june meeting, we will play around with a Raspberry Pi running FreeBSD and another with Arch Linux.





I'm planning on having two more in July, and 2 more in August etc. Until we've covered all of them. Well, almost. We will see.

So, what is *your* favorite OS on the Pi, and why?

François
@f_dion

I trace, U trace, DTRACE

That's the lightning talk that was selected by poll. That is going to be interesting to see how much info I can cram on dtrace and its use with Python in a quick lightning talk.

I'll see you at PYPTUG on monday night.


François
@f_dion

Pygame sound play

Música de fondo

En la lista Python tutor (en ingles) se hizo la pregunta de como añadir música en un vídeo-juego con pygame. Pueden leer mi respuesta en ingles en los archivos, o aquí en español. Es muy sencillo:

4 lineas, nada mas

Podemos hacerlo con python directamente en modo interactivo. La primera es incluir pygame.

>>> import pygame

Como es un juego pygame, es siempre presente, asi que en realidad hay que añadir solo 3 lineas mas.

>>> pygame.mixer.init(22050,-16,2,4096)

Las opciones del init, es según lo que uno desea (44100KHz o 22050KHz etc) o solo un init()

>>> snd = pygame.mixer.Sound("bach-cello-suite-1.wav")
>>> music = snd.play()

Ahora empieza la música. music.get_busy() nos dará un cero cuando es el fin de la música.

>>> music.get_busy()
1
>>> music.get_busy()
1
>>> music.get_busy()
0

Obviamente, el fichero .wav debe ser con el script .py o donde se ejecuta python.

François
@f_dion

dtrace: Python instrumentation

...where time becomes a loop

Last year, I mentionned that it was time for the Python community to embrace dtrace. I've gotten questions left and right, at user groups, through email etc as to what is dtrace and how it ties in with Python.

This week, a few posts on the Argentinian and Venezuelan Python lists on debugging Python and a total absence of a mention of dtrace and I knew I had to do a writeup. But before we get into the details, let's step back a bit.

Party like it's 1999 2004

Back in the 1990s I was using Povray (there is a Python API) to do photo quality rendering of made to order products. Eventually, I had to switch to OpenGL, C++, Sun Studio and hardware acceleration in order to keep up with the demand (over 20,000 during normal business hours and there are less than 30,000 seconds during that period of time). A few years later, at peak hours I was serving on the web over 100 renders per second.

Even if I had switched to OpenGL on that particular system, I continued working with Povray in other areas, particularly to design optical systems and build stuff that required visual quality over quantity. While Povray under Windows was fast enough, it felt much slower under Solaris and Linux (whereas my own code ran much faster on Solaris than Windows).

 

dtrace: First Contact

I posted the following to the solaris-x86 Yahoo group in November of 2004:

I finally ran Povray 3.5 benchmark 1.02 on the exact same hardware and here are the results:

Hardware:
Dell GX260
Pentium 4 2.4GHz
512MB ram
Hitachi 40 GB 7200 rpm
ATI Radeo 7500


Under Windows 2000 SP4, official Povray 3.5 win32 release:
Time for Parse: 2 seconds
Time for Photon: 54 seconds
Time for Trace: 36 min 47 seconds
Total time: 37 min 43 seconds


Under Solaris 10 B69, Blastwave Povray 3.5 x86
Time for Parse: 7 seconds
Time for Photon: 1 min 12 seconds
Time for Trace: 53 min 14 seconds
Total time: 54 min 33 seconds

Al Hopper suggested dtrace. I knew what it was (at the time, a Solaris only feature, now also available on Mac OS/X, FreeBSD, SmartOS, OpenIndiana and other IllumOS based OSes, and now Linux with dtrace4linux), but I hadn't taken the time to use it in real world cases. So I looked into it.

Instrumentation TNG

 

Which one is my blood pressure??

Here is what I posted back then:

I finally had a few minutes to play around with povray and dtrace this afternoon. I followed the suggestion made by Adam Leventhal on his Blog to run:

# dtrace -n 'pid$target:::entry{ @[probefunc] = count() }' -p <process-id>
(replace <process-id> by the pid of povray)

So what I did is run povray, get its process id with ps, then run the above. Once it rendered line 1, I ctrl-c. dtrace then spit out what I needed to know. I dont have any reference to compare as to what optimisation was done exactly on the windows build. However, running the above dtrace command on that process does reveal something:

I know it is spending a lot of time in DNoise / Noise, because it's called a bazillion times. Actually almost 10million times for the pair - the only other call that is called as much is memcpy, haven't investigated yet from where, but there might be an opportunity to combine memcpy. It also points out that a fast FSB and faster memory will definitely pull in front for equal cpu.

Anyway, back to Povray, looking at texture.cpp (line 169 and following):

/*****************************************************************************/
/* Platform specific faster noise functions
support */
/* (Profiling revealed that the noise functions can take up to 50%
of */
/* all the time required when rendering and current compilers
cannot */
/* easily optimise them efficiently without some help from
programmers!) */
/*****************************************************************************/

#if USE_FASTER_NOISE
#include "fasternoise.h"
#ifndef FASTER_NOISE_INIT
#define FASTER_NOISE_INIT()
#endif
#else
#define OriNoise Noise
#define OriDNoise DNoise
#define FASTER_NOISE_INIT()
#endif

Haha! Fasternoise.h (only found in the Windows source, not the Unix source) includes

/*****************************************************************************/
/* Intel SSE2
support */
/*****************************************************************************/

#ifndef WIN_FASTERNOISE_H
#define WIN_FASTERNOISE_H

#ifdef USE_INTEL_SSE2
int SSE2ALREADYDETECTED = 0 ;
DBL OriNoise(VECTOR EPoint, TPATTERN *TPat) ;
void OriDNoise(VECTOR result, VECTOR EPoint) ;
#include "emmintrin.h"
#include "intelsse2.h"
#undef ALIGN16
#define ALIGN16 __declspec(align(16))
#endif

#endif


BTW, with DTrace it only took me a few minutes total (including running povray for 3 minutes) to identify the culprit. Under windows it would have taken me hours.

So on Solaris x86, need to add USE_INTEL_SSE2 and USE_FASTER_NOISE as compile switches and add fasternoise and the various other includes from the windows source to the unix source.

And that is how a one liner dtrace script helped in debugging my performance problem with Povray. The nice thing is that you didn't even need the source to know what exactly what was going on in the code, in real time, without step by step debugging.

Instrumenting Python

So, what is the Python connection? When running a dtrace equipped version of Python, you can use dtrace on your Python scripts. On Solaris, this has been available since 2007 (in OpenSolaris).

Unfortunately on other OSes, it hasn't been the case. Jesus Cea has been trying to get John Levon's patches integrated with CPython since before 2.7 and 3.2 came out. Not sure what is needed to make this happen, but it is long overdue. At least some distro builders should have the patched Python available, but it really needs to be incorporated into the official build.

Anyway if you look at a Solaris 11 system, it has Python 2.6 as the default, and it is ready for dtrace:

 $ dtrace -lP python*  
   ID  PROVIDER      MODULE             FUNCTION NAME  
  1044 python1905 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  1045 python1905 libpython2.6.so.1.0           dtrace_return function-return  
  1046 python1939 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  1047 python1939 libpython2.6.so.1.0           dtrace_entry function-entry  
  1048 python1939 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  1049 python1939 libpython2.6.so.1.0           dtrace_return function-return  
  1050 python1945 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  1083 python7640 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  1084 python7640 libpython2.6.so.1.0           dtrace_return function-return  
  1100 python11695 libpython2.6.so.1.0           dtrace_entry function-entry  
  1101 python11695 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  1102 python11695 libpython2.6.so.1.0           dtrace_return function-return  
  1103 python11699 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  1214 python11693 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  1215 python11693 libpython2.6.so.1.0           dtrace_entry function-entry  
  1219 python11699 libpython2.6.so.1.0           dtrace_entry function-entry  
  1220 python11699 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  1221 python11699 libpython2.6.so.1.0           dtrace_return function-return  
  1226 python11693 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  1227 python11693 libpython2.6.so.1.0           dtrace_return function-return  
  1228 python11695 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  2248 python1905 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  2249 python1905 libpython2.6.so.1.0           dtrace_entry function-entry  
  2250 python23832 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  2251 python23832 libpython2.6.so.1.0           dtrace_entry function-entry  
  2260 python7640 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  2261 python7640 libpython2.6.so.1.0           dtrace_entry function-entry  
  2315 python23832 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  2316 python23832 libpython2.6.so.1.0           dtrace_return function-return  
  7670 python2936 libpython2.6.so.1.0           dtrace_entry function-entry  
  7671 python2936 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  7672 python2936 libpython2.6.so.1.0           dtrace_return function-return  
  7750 python14523 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
  7751 python14523 libpython2.6.so.1.0           dtrace_entry function-entry  
  7752 python14523 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
  7753 python14523 libpython2.6.so.1.0           dtrace_return function-return  
 12339 python1945 libpython2.6.so.1.0           dtrace_entry function-entry  
 12340 python1945 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
 12341 python1945 libpython2.6.so.1.0           dtrace_return function-return  
 12345 python2936 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
 12347 python1219 libpython2.6.so.1.0        PyEval_EvalFrameEx function-entry  
 12348 python1219 libpython2.6.so.1.0           dtrace_entry function-entry  
 12349 python1219 libpython2.6.so.1.0        PyEval_EvalFrameEx function-return  
 12350 python1219 libpython2.6.so.1.0           dtrace_return function-return  

The probes are specific to Python. In the original dtrace call I had used for debugging a C++ binary (Povray), I was using the pid provider and entry probe:

 # dtrace -n 'pid$target:::entry{ @[probefunc] = count() }' -p <process-id>  

But we are no longer in 2004 and I'm not interested in the performance of Povray right now, I just want to figure out how many checksums my pingpong.py script is doing (a little module to keep a tab on my machines and their latencies), and what else is going on at the top.

So, this time I'll modify it to use the python provider instead of the pid provider and using the function-entry probe (still doing a count): 

 # dtrace -qZn 'python$target:::function-entry{ @[copyinstr ( arg1 )] = count() }' -c ./pingpong.py  
 [...]  
  register                                                         10
  abstractmethod                                                   15
  __new__                                                          17
  <genexpr>                                                        35
  <module>                                                         48
  exists                                                           49
  S_IFMT                                                           53
  S_ISDIR                                                          53
  isdir                                                            56
  makepath                                                        100
  normcase                                                        100
  abspath                                                         113
  isabs                                                           113
  join                                                            113
  normpath                                                        113
  ping_pong                                                      1000
  checksum                                                       4000
  close                                                          4000
  do_one                                                         4000
  fileno                                                         4000
  receive_one_ping                                               4000
  send_one_ping                                                  4000
  __init__                                                       4007

With 1000 ping_pong() I was expecting 3000 checksums. Ah, I see I'm sending 4000 pings, so apparently I have a for that is not properly bounded (on purpose, to illustrate an example of what even something as simple as this can tell us).

To infinity and beyond

But that is not even touching the tip of the iceberg. How about creating heatmaps (node.js and dtrace in this case)? To do something like that with Python and dtrace, a good starting point is Brian Cantrill's Thijs Metsch's python-dtrace on Pypi (sorry for the wrong attribution, Brian's name was at the top of the page on Pypi).

You should also check out the following tutorial http://dtracehol.com/#Exercise_11 that was part of a tutorial session on dtrace at Java One last year (yep, Python at Java One).

François
@f_dion