Python + dtrace == insanely cool

dtrace is a sun developed (apple ported) technology for telling you what your computer is doing. I would try to explain it further but I would be doing it a great injustice. I first started learning about it here (http://www.youtube.com/watch?v=6chLw2aodYQ). Towards the end of the video Mr. Cantrill goes over using dtrace with python. One previous attempt at making this work on Leopard did not go over so well (but I did not put forth a lot of effort either) but giving it another try on Snow Leopard it seems to work just fine out of the box.

I downloaded the dtrace toolkit which has a lot of dtrace scripts. After unpacking it and navigating to the python directory I simply ran the following (at the command line).

sudo dtrace -s nameofscript.d’

I tried a bunch of the scripts provided in the tool kit and they work beautifully, providing an interesting look into what your python vm is up to.

Sphinx Documentation System

Sphinx is an excellent way of maintaining documentation, especially as it relates to programming, in particular python. I am a fan of the code highlighting and multiple output formats. I like using the html generation for day to day use but for the times I need more official looking documentation the pdf generation is excellent (if not a bit persnickety).

Installing sphinx is simple (easy_install sphinx) but it does not generate pdf’s out of the box, just latex. You can download latex for the mac from http://www.tug.org/mactex/. It is also not to difficult to install as it comes with an installer with the usual ./configure, make, sudo make install incantation.

Once all is installed you can start a sphinx documentation project from the command line with sphinx-quickstart. Adding and editing content is best described at Sphinx’s own website.

If you chose correctly in the setup you can now build documentation with a simple make command. Building pdf’s is a two step process where you ‘make latex’ and then go into the latex build directory where you ‘make all-pdf’. When all is said and done you should have a very presentable pdf file inside the latex build directory.

Like I mentioned before, I have found the pdf generation to be very sensitive to errors in your source files but they are “usually” easy to find and fix.

mssql + freetds + pyodbc + snow leopard

I develop on a mac but have to connect to mssql at some point or another. I had heard of freetds but I ended up taking the commercial way out with Actual Technologies mssql odbc driver for the mac because I wanted something that ‘just worked’ and it seemed reasonably priced. I have made do with that for over a year now.

My Snow Leopard upgrade rendered the Actual Technologies driver unusable with pyodbc and I am not overly sure why, something to do with 64bit even though I am not running the 64bit kernel. The driver itself would still talk to the database with it’s bundled ‘test connection’ but once coupled with pyodbc the house of cards came down. The support email request I sent out went unanswered so I started exploring alternatives while waiting.

I downloaded the latest freetds stable package from freetds.org. After snooping around online for a little while I found the configure script incantation that makes everything work (for me at least, your mileage may vary).

./configure –prefix=/usr/local/freetds –with-iodbc=/usr –with-tdsver=8.0
make
sudo make install

Specifying the tds version turned out to be the kicker for me to get it to work with pyodbc, without that freetds worked (tested using the bundled fisql tool) but did not work with pyodbc (not unlike Actual Technologies at this point). It seemed to ignore all tds version specification in the freetds.conf file (and I tried many from sources who swore they had a working installation). I found a random comment on sqlalchemy boards attesting to the same. I tried it out of desperation thinking that could not possibly work (that is why the conf file exists after all) but low and behold angels sung and my snow cat and python could finally talk to the evil empire database once again. One other thing I should note, I used the latest build of pyodbc available off github to test all this.

Snow Leopard does not for some odd reason come with the ODBC Administrator app that has accompanied previous versions of the 10.x series. It has been provided as a separate download by Apple. To finish the configuration I downloaded this app and went to the ‘Driver’ section and filled it out as depicted in the screen shot. One could optionally just create the correct config files in the correct locations and get the same result. I was unaware of all the correct locations and such until I found another web page detailing it so this was easier for me at the time.

FreeTDS Driver Config

With all this upgrading I decided to test out the two issues I filed about pyodbc and added comments to both. In summary, I would say both issues are resolved but I still am a bit confused about why unicode query strings do not seem to work.
- http://code.google.com/p/pyodbc/issues/detail?id=29
- http://code.google.com/p/pyodbc/issues/detail?id=46

Other resources:
- (wish I found this first) http://lists.ibiblio.org/pipermail/freetds/2009q2/024684.html
- http://cubist.cs.washington.edu/doc/FreeTDS/userguide/x1853.htm
- http://stackoverflow.com/questions/947077/using-pyodbc-on-linux-to-insert-unicode-or-utf-8-chars-in-a-nvarchar-mssql-field
- http://stackoverflow.com/questions/539430/connecting-to-ms-sql-server-using-python-on-linux-with-windows-credentials

Unicode is Evil

Unicode is evil, that is all.

Python Concurrency

Attended David Beazley’s python concurrency class over the weekend in Chicago. We were schooled on the horrific nature of the GIL, coroutines, generators, threads, processes, etc., etc. – a grand tour of nearly everything you can do to make multiple things happen at once. It was an excellent class and I would encourage anyone who wants to learn more about concurrency with python to attend future classes.

Mr. Beazley also has a very good collection of slides on coroutines/generators probably best described on the oubiwann blog but I will list here as well.

• http://www.dabeaz.com/coroutines/index.html
• http://www.dabeaz.com/generators-uk/index.html

From the class a concurrency mailing list has sprung up on google groups: http://groups.google.com/group/python-concurrency

pyodbc and ubuntu (jaunty)

Mr. Eden has a great post on accessing mssql from linux.

I had to add one package (unixodbc-dev) to get things working so I ended up with the following:

sudo aptitude install unixodbc unixodbc-dev freetds-dev tdsodbc python-dev


Once this was done the usual “python setup.py install” no longer ended in tears. He also included instruction on editing the odbcinst.ini file.

memcache on ubuntu (intrepid ibex)

Took me a few minutes to find all this so I thought I should take note.

Installing memcache
sudo apt-get install memcached

This installs memcache in the usual ubuntu/debian way. It scatters several important files around the operating system in the process.

• start script: /usr/share/memcached/scripts/start-memcached
• init.d startup: /etc/init.d/memcached
• config file: /etc/memcached.conf

Need to edit the conf file as by default it only lists on the localhost.

Below are some memcache links for more reading:

• http://www.danga.com/memcached/
• http://code.google.com/p/memcached/wiki/TutorialCachingStory -> answers the question of how to explain memcache to others
• http://lzone.de/articles/memcached.htm -> testing/hacking memcache with telnet

David Beazley’s Introduction to Python Class

Attended a Python class this week.  Mr. Beazley put’s on a great introduction to python class and don’t let the “Introduction” fool you, it is a course with “meat”.  Highly recommended to anyone who already programs and is interested in learning python.

His site: http://www.dabeaz.com/

The class is also offered at the Big Nerd Ranch (which is a great place for training in my experience): http://www.bignerdranch.com/classes/python.shtml

My first time with Ruby and Gem

I had no real opinion on ruby.  Heard it was like python, just different.  I like python.  I wanted to take a look at sproutcore, new javascript library that all the cool kids are talking about.  sproutcore uses ruby for a variety of reasons. Whatever, no big.  I work on a mac so Leopard has ruby preinstalled right?.  A quick check proved my suspicions.

ruby -v 

ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]

gem -v

1.0.1

Ok, so maybe I will just install sproutcore (using gem which seems to be ruby’s easy_install) and go on to see what all this brand new (yet just another) javascript library fuss is about.  

sudo gem install sproutcore

This gave me …

Bulk updating Gem source index for: http://gems.rubyforge.org

I killed it after approximately 30 minutes (that is not a misprint, I said 30 MINUTES) believing it perhaps did not understand I just wanted to fracking install sproutcore not download/update the index of the ruby gods.  Just sproutcore please.  

I tried it again, same results but my patience was on a much shorter fuse.  Tried the –help to see if there was a flag to set gem to not fuckall but none was apparent to me. 

Not willing to give up that easily I went to http://gems.rubyforge.org.  Simple website which informed me I could see the index in yaml.  Ok.  It downloaded in 4-5 seconds.  Is this yaml document the same index that gem is speaking of?  If so what the hell is gem doing?  

So anyways … fuck ruby and gem.

PostgreSQL with some Jeos

PostgreSQL – Open source database (their website here).

Jeos – Stripped down Ubuntu for Virtual Machines (tutorial here).

I need a custom build of postgres because I have to get it to work with Windows Active Directory which from the literature (blogs) I have read works.  I am somewhat skeptical but PostgreSQL will be a hard sale unless it does what the literature says so I am off to test the claims.

I like to start many tests on Ubuntu’s Jeos precisely because of its stripped down nature.  I eliminate a lot of possible cruft upfront.  This present’s some problems when building software like PostgreSQL.  After following the steps in the Jeos tutorial of course it would be easy to say:

sudo apt-get install postgresql postgresql-client 

This would accomplish the task just fine but since I want a custom build it is a no go.  Just running the make script in the source distribution has issues too because PostgreSQL has more than it’s fair share of dependencies.  Enter apt’s build-dep.  Someone really thought this through for me already.  Bless you Debian and Ubuntu.  Working with tools like these is like a breath of fresh air after a day of working with the a great commercial “Enterprise” system.  It is true that people really exist who actually put thought into their products instead of a marketing website.  End Rant.  With a simple command you can get a list of dependencies that need to be built and installed for PostgreSQL (the -s means to not actually install anything, just a simulation).

sudo apt-get -s build-dep postgresql postgresql-client

Look over the list if you desire and now just take out the -s to actually do the build and install of all the dependencies that you need.

Next step, build PostgreSQL.  Coming soon.

Other Sources:

http://www.debian.org/doc/manuals/apt-howto/ch-sourcehandling.en.html

Directed Acyclic Graph

Directed Acyclic Graph.  Big Words, dag for short?  Interesting data model for the tool box.

Git uses this data structure to keep track of it’s repository, branches, etc.

• http://en.wikipedia.org/wiki/Directed_acyclic_graph
• http://eagain.net/articles/git-for-computer-scientists/

I have been representing a lot of data in a tree format as of late.  This data model looks like an interesting substitute… more general and flexible than the simple tree’s I use.  Something to read more about.

Leopard Kerberos Version

What version of kerberos is leopard running?  

This took a bit of googling to answer so I thought I would take a note.

Apple is not very helpful, they say I am running:

Kerberos        KfM-6.0fc5 
Kerberos Administration System  KfM-6.0fc5 

Which once tracked down, here and here, Leopard is running MIT Kerberos Version 1.6 

Finally, the prompt for all this.

Does the file exist?

 

Python 2.5 has a cool any(iterable) function.  I needed to know if a file existed somewhere in a tree of files.  With any you can write something like the following:

    dirs = os.walk(os.path.abspath(''))  
    fileExists = any('testfile.txt' in x[2] for x in dirs)

Now the variable fileExists will be True if the file testfile.txt exists anywhere in the tree. 

two lines.  python is neat.

Connection Strings

odbc and mssql are the least documented/supported databases in nearly all open source projects.  I understand, evil empire, i get it.  So when I dig up gems after tons of googling I should write them down so …

Connection String for pyodbc, using a system DSN

"DSN=dsn_name;UID=username;PWD=passwd"

Connection String for sqlalchemy, using a system DSN

'mssql://username:passwd@/?dsn=dsn_name'

Leave the @ symbol after the passwd and don’t use an @ symbol in your passwd (guilty). I assume the symbol is used in the parsing of the string, why I am not so sure on.

An Alchemist’s Trials

SQL Alchemy is a terrific little library.  At least I think so on days ending in p. To be quite honest the whole ORM thing frightens me a little, too many layers between where your using your data and where it is being stored. Layers are not inherently bad in my opinion but they can drive you bananas debugging problems if you don’t understand what those layers are doing. I do not understand what SQL Alchemy is doing. Hence why I am going b-a-n-a-n-as.

But when I am not using an ORM I find my self writing half assed “database managers” which have a tendency to grow over time into unwieldy beasts.  So when a new project cropped up I decided to take the plunge, find an ORM library, learn it, love it.  I shopped around in the Python world, for it was the chosen language for the new project, and SQL Alchemy’s name kept coming up. O’reilly was about to publish a book on it which for me is like the project is getting a blessing from the gods (I like books and O’reilly has cute wild life pictures on the covers).

So I hope to take down some helpful notes on alchemy related goings on so I have a reference to all this insanity.

mssql – odbc – python hell

I am using python. Hooray for new fangled scripting glory. 
My business is an mssql shop.  meh.
I set up pyodbc. I run a simple insert statement. Sql Server Profiler logs the following (among other tidbits).

declare @p1 int
set @p1=1
exec sp_prepare @p1 output,N'@P1 VARCHAR(11)',N'insert into schemax.iptable (IPAddress) values(@P1)',1
select @p1

Very simple insert me thinks.
I want to try new orm hotness that is SQL Alchemy.
Again with the very simple test. SQL Server Profiler logs the following:

declare @p1 int
set @p1=13808
exec sp_prepare @p1 output,NULL,N'',1
select @p1

See the subtle difference?

No errors are ever thrown. It runs its little heart out. But no data is every actually inserted into the database.

I loathe odbc.  

Also, if your ever perusing the sparse documentation available on pyodbc, and see all those helpful quick examples be sure to call the commit() method which they do not show or set the autocommit flag to true.

update: my woes on the sqlalchemy-devel list here.