Sighup & Python

I wrote a python processing daemon on a machine, that was supposed to run indefinitely which used to terminate mysteriously by itself. It was later traced to dying on logout, because I generally leave a machine terminal logged into the remote machine.

Later on, I checked if it was the notorious SIGHUP, a hangup signal issued to all processes when the user logs off.

In order to ascertain this, I used the following line to send a sighup signal to the process


sudo kill -SIGHUP 2025


You can get a list of process details(also containing the process id’s)


ps aux | grep python mydaemon.py


Now there are multiple ways of how to handle this:-

1) Using NOHUP:


sudo nohup python mydaemon.py


The problem with this approach is that I was calling system commands from my code which themselves were placed in /usr/bin. After using nohup to run the daemon, this access was restricted and my code threw exceptions saying that the commands were not found.

2) Using upstart:

This is a Ubuntu based daemon manager. It has all these features out of the box, sadly I havent had the time to use it. Its available on Redhat and other distros of linux as well.

3) Python:

Implementing a signal handler. This is by far the most convinient.


import signal
signal.signal(signal.SIG_HUP, signal.SIG_IGN)


SIG_IGN is a default signal handler that will do nothing. You could alternatively implement your own handler for various OS Signals in the following way:-


def handler(signum, frame):
print 'Did anyone HICHUP ?'
signal.signal(signal.SIG_HUP, signal.SIG_IGN)


If anyone has any better more professional ways of doing this, do let me know.

SVN 1.4.3 to 1.4.6 upgrade issue

One of our old faithful deployment machines at work got reinstalled and we upgraded SVN from version 1.4.3 to 1.4.6

However the data was incompatible, so we had to run the following commands in order:-

for every folder:-

Take a dump of that path and redirect the output to a temporary path and preserve the relative heirarchy.
Delete the Files under the Path
ReCreate the following frompath which just got deleted
Load this frompath with data from the temporary File Path

And thats it. It can all be done using svncopy.py Since I dont have an online CVS or SVN account, I’ll paste the code here.

I recommend you to look at the code and comment out the lines that you want to comment out. I bear no responsiblity in case anything goes wrong. Please backup the folder if you fear data loss.

"""
Disclaimer: The following code is being released to the public with no strings attached.
However I do not take responsiblity in any way for the side effects of using this code.

Contributed by : Afrobeard
"""

import os;
import dircache;
from optparse import OptionParser;

def fixpath(path):
path = path.replace('\\', '/').replace('//', '/');
path = path.replace('\\', '/').replace('//', '/');

if path == '.':
pass;
elif path == './':
path = '.';
elif path.startswith('./'):
path = path[2:]
if path.startswith('/'):
path = path[1:];
if path.endswith('/'):
path = path[:-1];
return path;

def runcmd(cmd):
print cmd
os.system(cmd)

parser = OptionParser()
parser.add_option("-f", "--from", dest="frompath",
help="From Path. Will be deleted later on")
parser.add_option("-t", "--to", dest="topath",
help="To Path.")

(options, args) = parser.parse_args()

options.frompath = fixpath(options.frompath)
options.topath = fixpath(options.topath)

os.chdir(options.frompath)

dumpcmd = 'svnadmin dump "'+options.frompath+'/%s" > "'+options.topath+'/%s"'
rmcmd = 'sudo rm -Rf "%s"'
createcmd = 'svnadmin create "' + options.frompath + '/%s"'
loadcmd = 'svnadmin load"' + options.frompath + '/%s" 2:
pass;
elif len(os.path.join(root,name).split('/')) > 2:
pass;
else:
print os.path.join(root,name)
print name

#1. Taking the DUMP
runcmd(dumpcmd % (name, name))

#2. Removing the from folder
runcmd(rmcmd % name)

#3. Creating the Path into the repository
runcmd(createcmd % name)

#4. Loading contents
runcmd(loadcmd % (name, name))

PyUnicodeUCS2_DecodeUTF8

The old centos 4 installation at work had been running fine, until the day I accidentally deleted python2.3 off of it. I needed to install python2.5 and didnt think this would cause any problems. However I ended up breaking yum [Our friendly package manager].

Quite a few people tried to fix this by installing 2.3 and although I cant give a detailed series of actions that led to this, but we were getting problems with PyUnicodeUCS2_DecodeUTF8 or more specifically


undefined symbol: PyUnicodeUCS2_DecodeUTF8


A little research into te problem indicated that its because of a representation issue for Unicode. The modules in question are using 4 bytes to represent a unicode whereas my Version of Python was using 2.

http://effbot.org/pyfaq/when-importing-module-x-why-do-i-get-undefined-symbol-pyunicodeucs2.htm

Quotes that:-

The only way to solve this problem is to use extension modules compiled with a Python binary built using the same size for Unicode characters.

However I was getting this inside pythons own libraries.

Therefore I Went to http://rpm.pbone.net and Went to Advanced rpm search, selected CentOS 4, and got rpm-python for the selected platform.

I Downloaded the package via wget and then ran the following command to unpack the rpm file:-


rpm2cpio python-2.3.4-14.4.i386.rpm | cpio -idmv


This made a heirarchy inside my working folder starting with the relative path usr.

After that, I copied everything from here to /usr/lib/python2.3 [or the Lib folder for python.]

I fired up yum and Everything worked like a breeze

[x for x in list while condition]. Alternatives for Python

Recently I came across the following comp.lang.python usenet thread. urikaluzhny posted the following question:-

It seems that I rather frequently need a list or iterator of the form [x for x in <list> while <condition>]
And there is no one like this.
May be there is another short way to write it (not as a loop).

I think Paul Rubin gave a reasonable answer in which he said to use itertools.takewhile(condition, seq)

Other solutions were bigger ones i.e. the long way to write it.

I was thinking of the shortest way to do this, using pure python constructs. i.e without importing anything addional and I came up with the following:-


l = range(6)
1 if b!=4 else l.__delslice__(0,len(l)) for b in l][:-1]


I was wondering if anyone could come up with something more concise no matter how complex. Feel free to respond on the original thread or here[In which case I'll shift the answers I consider reasonable onto the original thread using your name of course]. Or you could feel free to post at both places.

Disclaimer:-

1. The following proposed solution is not intended to be a solution, it goes completely against the zen of python. [Type import this into the python command interpreter]
2. It is strongly undesireable for us humans to use anything that starts with __

Are your PDF’s not being Rendered Correctly

Its probably because of a bad PDF definition. A bad PDF definition constitutes of but is not totally comprised of Font embedding. Other errors can consist of messed up gamma values, etc which may result in the pictures looking different when viewed with different PDF Readers but lets not dwelve into those at the moment.

So how exactly is one supposed to embed fonts into an Acrobat pdf. If you’re using Adobe Acrobat Distiller, instructions can be found here. In case you’re using GhostScript, you can find help here. It can be implied that the types of font one should use should be embeddable, e.g. TrueType or Type1.

At this point you may be wondering about what to do if you have a document that does not have fonts embedded into it. Generally PDF Viewers can be supplied with font file paths so that they can pick font definitions from there. So you just need a comprehensive collection of fonts on your system.

Where does one get a collection of fonts to use. Printers use postscript to define documents and they contain a collection of common fonts to render page output correctly. I extracted some fonts out of some Lexmark printer drivers once. I’ve uploaded them and they can be downloaded here.

In ghostscript, the font file directory can be specified as follows[following extract copied blatantly from here]:-

1) Symbolic link to the font directory as the expected dir for example:

ln -s /usr/local/share/ghostscript/fonts app/ghostscript-8.00/share/ghostscript/fonts

2) Tell gs where to find the fonts using the -I option to add a search path. For example:

gs -I/usr/local/share/ghostscript/fonts …

3) Set the GS_LIB environment variable to specify the dir:

export GS_LIB=/usr/local/share/ghostscript/fonts

Please let me know in the comments if any of these links particularly to the compressed fonts file dies.

Unicode & MySQLdb

I dont like giving cursory descriptions about problems, but I got plagued with the following error message when using xmppp to send messages after database lookups:-


File "D:\Python25\Lib\site-packages\xmpp\protocol.py", line 418, in __init__
if body: self.setBody(body)
File "D:\Python25\Lib\site-packages\xmpp\protocol.py", line 431, in setBody
self.setTagData('body',val)
File "D:\Python25\Lib\site-packages\xmpp\simplexml.py", line 243, in setTagDat
a
except: self.addChild(tag,attrs,payload=[ustr(val)])
File "D:\Python25\Lib\site-packages\xmpp\simplexml.py", line 32, in ustr
if type(r)type(u''): return unicode(r,ENCODING)tBody
self.setTagData('body',val)
File "D:\Python25\Lib\site-packages\xmpp\simplexml.py", line 243, in setTagDat
a
except: self.addChild(tag,attrs,payload=[ustr(val)])
File "D:\Python25\Lib\site-packages\xmpp\simplexml.py", line 32, in ustr
if type(r)type(u''): return unicode(r,ENCODING)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 20-21: invalid data


I was trying to convert the data being received by the sql connector to a unicode string and it was giving invalid data exceptions. The tables contained Unicode fields and I was retreiving unicode however the data coming from the connector were normal python ASCII Strings. Some other errors I saw while playing around with my code included:-


UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
52: ordinal not in range(128)


The solution was simple. Rather than using


self.conn = MySQLdb.connect(host=self.host,
user = self.user,
db = self.db)


I had to use


self.conn = MySQLdb.connect(host=self.host,
user = self.user,
db = self.db,
use_unicode = True, charset="utf8"
)


And viola things worked seamlessly eversince and I didnt need to do any conversion.

I didnt find the time to investigate more on what was actually happening in terms of how the data was actually being stored. I hope this fix was good enough for most, but for the more technically inclined, maybe I’ll investigate further and update this article in the future.

Flight of the Conchords : The humans are dead

Python’s equivalents to the Ternary operator cond?then:else

Recently a friend of mine asked me if there was a ternary operator in Python and it got me thinking.

Languagues such as C even have a syntax like [[ condition ? ifblock : thenblock ]]

The first thought that came to mind was using Dictionaries and lambda functions. A dictionary can also use the Boolean type for true so:-

{‘True’: lambda x : ifblockfunction(), ‘False’: lambda x : elseblockfunction()}[condition]()

So basically its a dictionary lookup agaist the condition that returns the apprpriate lambda function. Since lambda’s are functions, the () at the end is that of a function call.

Okay you guys still with me, cos I have more to tell you.

I learned python over a year ago back in the days of 2.4 and a notation that I hadnt come across that exists from 2.5 onwards is the [[ ifblock if cond else elseblock ]].

>>>5 if False else 7

Should return 7.

Pretty nifty I’d say, resembles what its parse tree would look like.

If you’re using python 2.4 or below, you must be quite disappointed by now but theres more, however you’ll have to bear with me on this one.

In python, datatypes inside an if statement evaluate to True or False. So if(3) should give True[Note not actual language syntax]. Similarly if([1,3,4]), if ((1,3,4)), etc all return True.

So what returns false. None, False, empty lists, empty tuples, etc return false

This is a functionality that Python could have inherited LISP since in LISP everystatement evaluates to something.

1234 and True

The above line should evaluate to True which it does. But consider this

>>> True and 1234
1234
>>> 1234 and True
True

True and 1234 evaluates to 1234, so its returning the second True Element and it doesnt make a differece because essentially both True and 1234 are True

So how can we use this to our liking. Consider the following.

“T” and condition or “F”

This would return “T” for True and “F” for false by shortcircuit evaluation.

However func1() and True or func2() wont work.

Function calls return None to the language and func1() would be executed anyways during the inorder execution so it would be of no use to us.

Therefore in order to make this work we can once again resort to lambda functions as follows:-

>>>(lambda : mojo()) and False or (lambda : len(“sdf”))()
3

This is the geekiest solution by far and it has the same inorder tree placement as its syntax in 2.5

P.S. I love language related information. If anyone found this useful and has anything to share, gems left in the comments section for me to munch on shall highly be appreciated.

Sicko, A must-watch if you’re from, in or interested in the US

I recently watched Sicko, a documentary directed by Michael Moore about the American health care system. The problem with the US is that since its such a highly service oriented industry, they dont try to give him a maximum return on his money if there is no competition, rather the parts of your body are divided by function. An example that which deeply effected me from the movie was that of a carpenter who lost both his index and his ring fingers in a carpentry accident. The cost of reattaching the fingers[for the carpenter] varied with the functionality of the fingers rather than the price it cost the hospital to reattach them. Could there be a system more evil than one who’s only purpose is to maximize profits. In a country such as the US, health care should be the responsibility of the government and should not be provided by private companies.

The film is full of heart warming anecdotes, facts & comparisons that were the result of a through study of the american health care system. I wouldnt want to leak too many details, but this is a must watch.

I also watched the Corporation which is also an interesting film. According to some experts FOSS costs the software industry 70 billion US dollars ever year in missed profits and sales oppurtunity which means that eventually large companies would have to change the way they operate. And these developments are good for the little guys out there because a free trade system with a lot of small companies and individuals offering services is a much better option in my opinion.

Good Bye Vista

You sha’nt be missed. If you guys were wondering what Vista was doing on my system in the first place, I shall confess to err is human. But I have learnt from my mistakes and I shall follow the right path now.

I shall start this post by listing issues I faced with Windows Vista:-

1. Blue Screens of Death. Lots of em. Reminded me of the days of Win 95.
2. Weird Graphical Glitches. Sometimes my screen looks like two screens shifted by a couple of pixels and overlapping each other.
3. Games running too slow. Well I’ll give you the example of HellGate london. I can run it on full specs 1920×1200 on my new XP installation, however on vista I had to run it on 1080×1024 on my laptop [[2GB Ram Nvidia 7950 GTX Core2Duo Santa Rosa Architecture]]
4. Too much clicking. In attempts to make everything look better, they’ve included in a lot of clicks. In XP you could right click Network Icon -> Status -> Properties and change your network config. In Vista however you have you open Network Center.
5. Ctrl+Alt+Delete < This is quite significant. Rather than bringing up the Task Manager directly in vista you have to see a fancy splash screen first.
6. Games not running. Yep its true. There are major incompatibility issues to be faced here.

And the best one which actually caused me to get rid of vista. They burn some keys into the MBR so that it isnt visible if someone installs another MBR. It has been known to wreak havoc on dual booter systems. I discovered this while installing FreeBSD on my system. Thank you for your Guidance Richard Stallman.

In case you have been hit by this problem and are seeking remedial, you can move over to http://www.clearchain.com/wiki/FreeBSD_&_Windows_Vista  for a solution.  If microsoft paired CD keys for Vista with XP, I’m sure there’d still be 75% peopleof this population still using XP.