Python Concurrency

Attended David Beazley’s python concurrency class over the weekend in Chicago. We were schooled on the horrific nature of the GIL, coroutines, generators, threads, processes, etc., etc. – a grand tour of nearly everything you can do to make multiple things happen at once. It was an excellent class and I would encourage anyone who wants to learn more about concurrency with python to attend future classes.

Mr. Beazley also has a very good collection of slides on coroutines/generators probably best described on the oubiwann blog but I will list here as well.

• http://www.dabeaz.com/coroutines/index.html
• http://www.dabeaz.com/generators-uk/index.html

From the class a concurrency mailing list has sprung up on google groups: http://groups.google.com/group/python-concurrency

pyodbc and ubuntu (jaunty)

Mr. Eden has a great post on accessing mssql from linux.

I had to add one package (unixodbc-dev) to get things working so I ended up with the following:

sudo aptitude install unixodbc unixodbc-dev freetds-dev tdsodbc python-dev


Once this was done the usual “python setup.py install” no longer ended in tears. He also included instruction on editing the odbcinst.ini file.

git things

Cultured code has a nice little product called Things. Problem is that I live on multiple machines and Things does not support syncing as of yet. They helpfully suggest putting your things library on a usb stick. This might work but for the fact that I hate carrying around usb drives (they are small and easily forgotten, misplaced, etc).

So instead I am placing the Things library inside a git repository. A git pull here and a git push there (here a git, there a git, everywhere a git git) and it all syncs up, minus the bad pun.

memcache on ubuntu (intrepid ibex)

Took me a few minutes to find all this so I thought I should take note.

Installing memcache
sudo apt-get install memcached

This installs memcache in the usual ubuntu/debian way. It scatters several important files around the operating system in the process.

• start script: /usr/share/memcached/scripts/start-memcached
• init.d startup: /etc/init.d/memcached
• config file: /etc/memcached.conf

Need to edit the conf file as by default it only lists on the localhost.

Below are some memcache links for more reading:

• http://www.danga.com/memcached/
• http://code.google.com/p/memcached/wiki/TutorialCachingStory -> answers the question of how to explain memcache to others
• http://lzone.de/articles/memcached.htm -> testing/hacking memcache with telnet

David Beazley’s Introduction to Python Class

Attended a Python class this week.  Mr. Beazley put’s on a great introduction to python class and don’t let the “Introduction” fool you, it is a course with “meat”.  Highly recommended to anyone who already programs and is interested in learning python.

His site: http://www.dabeaz.com/

The class is also offered at the Big Nerd Ranch (which is a great place for training in my experience): http://www.bignerdranch.com/classes/python.shtml

My first time with Ruby and Gem

I had no real opinion on ruby.  Heard it was like python, just different.  I like python.  I wanted to take a look at sproutcore, new javascript library that all the cool kids are talking about.  sproutcore uses ruby for a variety of reasons. Whatever, no big.  I work on a mac so Leopard has ruby preinstalled right?.  A quick check proved my suspicions.

ruby -v 

ruby 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]

gem -v

1.0.1

Ok, so maybe I will just install sproutcore (using gem which seems to be ruby’s easy_install) and go on to see what all this brand new (yet just another) javascript library fuss is about.  

sudo gem install sproutcore

This gave me …

Bulk updating Gem source index for: http://gems.rubyforge.org

I killed it after approximately 30 minutes (that is not a misprint, I said 30 MINUTES) believing it perhaps did not understand I just wanted to fracking install sproutcore not download/update the index of the ruby gods.  Just sproutcore please.  

I tried it again, same results but my patience was on a much shorter fuse.  Tried the –help to see if there was a flag to set gem to not fuckall but none was apparent to me. 

Not willing to give up that easily I went to http://gems.rubyforge.org.  Simple website which informed me I could see the index in yaml.  Ok.  It downloaded in 4-5 seconds.  Is this yaml document the same index that gem is speaking of?  If so what the hell is gem doing?  

So anyways … fuck ruby and gem.

PostgreSQL with some Jeos

PostgreSQL – Open source database (their website here).

Jeos – Stripped down Ubuntu for Virtual Machines (tutorial here).

I need a custom build of postgres because I have to get it to work with Windows Active Directory which from the literature (blogs) I have read works.  I am somewhat skeptical but PostgreSQL will be a hard sale unless it does what the literature says so I am off to test the claims.

I like to start many tests on Ubuntu’s Jeos precisely because of its stripped down nature.  I eliminate a lot of possible cruft upfront.  This present’s some problems when building software like PostgreSQL.  After following the steps in the Jeos tutorial of course it would be easy to say:

sudo apt-get install postgresql postgresql-client 

This would accomplish the task just fine but since I want a custom build it is a no go.  Just running the make script in the source distribution has issues too because PostgreSQL has more than it’s fair share of dependencies.  Enter apt’s build-dep.  Someone really thought this through for me already.  Bless you Debian and Ubuntu.  Working with tools like these is like a breath of fresh air after a day of working with the a great commercial “Enterprise” system.  It is true that people really exist who actually put thought into their products instead of a marketing website.  End Rant.  With a simple command you can get a list of dependencies that need to be built and installed for PostgreSQL (the -s means to not actually install anything, just a simulation).

sudo apt-get -s build-dep postgresql postgresql-client

Look over the list if you desire and now just take out the -s to actually do the build and install of all the dependencies that you need.

Next step, build PostgreSQL.  Coming soon.

Other Sources:

http://www.debian.org/doc/manuals/apt-howto/ch-sourcehandling.en.html

Directed Acyclic Graph

Directed Acyclic Graph.  Big Words, dag for short?  Interesting data model for the tool box.

Git uses this data structure to keep track of it’s repository, branches, etc.

• http://en.wikipedia.org/wiki/Directed_acyclic_graph
• http://eagain.net/articles/git-for-computer-scientists/

I have been representing a lot of data in a tree format as of late.  This data model looks like an interesting substitute… more general and flexible than the simple tree’s I use.  Something to read more about.

Leopard Kerberos Version

What version of kerberos is leopard running?  

This took a bit of googling to answer so I thought I would take a note.

Apple is not very helpful, they say I am running:

Kerberos        KfM-6.0fc5 
Kerberos Administration System  KfM-6.0fc5 

Which once tracked down, here and here, Leopard is running MIT Kerberos Version 1.6 

Finally, the prompt for all this.

Does the file exist?

 

Python 2.5 has a cool any(iterable) function.  I needed to know if a file existed somewhere in a tree of files.  With any you can write something like the following:

    dirs = os.walk(os.path.abspath(''))  
    fileExists = any('testfile.txt' in x[2] for x in dirs)

Now the variable fileExists will be True if the file testfile.txt exists anywhere in the tree. 

two lines.  python is neat.

Connection Strings

odbc and mssql are the least documented/supported databases in nearly all open source projects.  I understand, evil empire, i get it.  So when I dig up gems after tons of googling I should write them down so …

Connection String for pyodbc, using a system DSN

"DSN=dsn_name;UID=username;PWD=passwd"

Connection String for sqlalchemy, using a system DSN

'mssql://username:passwd@/?dsn=dsn_name'

Leave the @ symbol after the passwd and don’t use an @ symbol in your passwd (guilty). I assume the symbol is used in the parsing of the string, why I am not so sure on.

An Alchemist’s Trials

SQL Alchemy is a terrific little library.  At least I think so on days ending in p. To be quite honest the whole ORM thing frightens me a little, too many layers between where your using your data and where it is being stored. Layers are not inherently bad in my opinion but they can drive you bananas debugging problems if you don’t understand what those layers are doing. I do not understand what SQL Alchemy is doing. Hence why I am going b-a-n-a-n-as.

But when I am not using an ORM I find my self writing half assed “database managers” which have a tendency to grow over time into unwieldy beasts.  So when a new project cropped up I decided to take the plunge, find an ORM library, learn it, love it.  I shopped around in the Python world, for it was the chosen language for the new project, and SQL Alchemy’s name kept coming up. O’reilly was about to publish a book on it which for me is like the project is getting a blessing from the gods (I like books and O’reilly has cute wild life pictures on the covers).

So I hope to take down some helpful notes on alchemy related goings on so I have a reference to all this insanity.

mssql – odbc – python hell

I am using python. Hooray for new fangled scripting glory. 
My business is an mssql shop.  meh.
I set up pyodbc. I run a simple insert statement. Sql Server Profiler logs the following (among other tidbits).

declare @p1 int
set @p1=1
exec sp_prepare @p1 output,N'@P1 VARCHAR(11)',N'insert into schemax.iptable (IPAddress) values(@P1)',1
select @p1

Very simple insert me thinks.
I want to try new orm hotness that is SQL Alchemy.
Again with the very simple test. SQL Server Profiler logs the following:

declare @p1 int
set @p1=13808
exec sp_prepare @p1 output,NULL,N'',1
select @p1

See the subtle difference?

No errors are ever thrown. It runs its little heart out. But no data is every actually inserted into the database.

I loathe odbc.  

Also, if your ever perusing the sparse documentation available on pyodbc, and see all those helpful quick examples be sure to call the commit() method which they do not show or set the autocommit flag to true.

update: my woes on the sqlalchemy-devel list here.