Things I hate about Python/Numpy/Scipy

Well recently reading the great post
http://use.perl.org/~brian_d_foy/journal/32556?from=rss has inspired me to get
off my chest things about python/numpy/scipy that I hate. (With the
understanding that I love Python for science, and use it almost exclusively.
But if I had a million dollars and infinite influence I would fix these).

1. Keyword arguments are parsed in as a dictionary, and are therefore unordered.
   This makes it a constant paint to use nice syntax to define ordered types. Say
   for example I want a list like object with element names (a la R) in a perfect
   world I would be able to say NList(a=1, b=2, c=3) and expect the order of the
   resulting list to mimic the order of the keyword definitions (again like R
   does). Instead I need to use NList((‘a’, 1), (‘b’, 2), (‘c’, 3)) which is
   plain ugly, and pedantic. Worse in the new OrderedDict type of python 2.7/3.1
   you can use keyword arguments and get arbitrary order! This seems like a big
   interface wart, it sould not have been allowed.
2. NumPy/SciPy’s interfaces are painfully inconsitent. At best they mimic
   Matlab, which is a bad idea as Matlab’s argument unpacking is more powerfull
   than Python’s, and with objects it is suboptimal in any case. Mathematica and
   R/S++ are much better models for what a powerfull programming language means
   for scientific interfaces.
3. The BSD license requirement for NumPy/SciPy. I understand the arguments for
   this, but I find it painfull to watch libraries being rewritten from scratch
   when amazing GPL equivalents exist. Python for science is made worse for this.
   R, Octave, and GSL are giant communities that the Python world, in my opinion,
   will waste all of its developers trying to reproduce in a BSD way.
4. Numpy array’s have too many methods, such as std, var, cov which should be
   functions instead. An ndarray should just be a fast multidimensional object
   that can be reshaped, accessed, and operatered on. If a richer object with
   statistical functions is needed it should be a seperate subclass, as this adds
   bloat and often is not needed in the context that ndarray’s are used. Further
   the default behavior of std uses the population formula instead of the de
   facto standard of the sample formula (used by every other scientific package
   – ever). Some developers see this as a needed revolution, but as a method to
   an extension type it can not be easily overrided (without a serious slowdown
   in all method calls) which makes for the annoying code of .std(ddof=1) all the
   time, unless you breakdown and use use the (which would require explaination
   in almost any paper) or warn people to not use the method!.
5. Finally a minor gripe that I covet from R, but don’t think can be added in
   a pythonic manner is the ability to use values in default arguments from other
   default arguments. That is something like size(n=10, step=n/2). In python we
   would need to use extensive None defaults and handle the logic in the function
   which removes documentation.

Well that feels better.

Python Commandlets Redux

Reading the awesome python blog word aligned I was directed to the cool python code snippet at voidspace that creates a simple class for making unix shell like pipes for functions in python. Thinking this was cool, I ported over some of the Ironpython specific features to be CPython compatible, and used the classes as decorators as I think this makes the code more direct.

The compatibility parts where to replace ‘System.DateTime.Parse’ to the dateutil function ‘parse’, which returns a datetime object and not a float like the IronPython call. So I then updated the Path object to store datetime objects for ‘mtime’ and ‘ctime’, which I think is nicer anyway. The Full code is:

import os
from dateutil.parser import parse
from datetime import datetime
from cmdlet import Cmdlet, Action

class Path(object):
    def __init__(self, path, entry):
        self.dir = path
        self.name = entry
        self.path = os.path.join(path, entry)
        self.mtime = datetime.fromtimestamp(os.path.getmtime(self.path))
        self.ctime = datetime.fromtimestamp(os.path.getctime(self.path))

    def __repr__(self):
        start = 'File:'
        if os.path.isdir(self.path):
            start = 'Dir:'
            ctime = self.ctime
            mtime = self.mtime
            return "%s %s :ctime: %s :mtime: %s" % (start, self.path, ctime, mtime)

@Cmdlet
def listdir(path):
    def listdir():
        return [Path(path, member) for member in os.listdir(path)]
    return listdir

@Cmdlet
def notolderthan(date):
    datetime = parse(date)
    def notolderthan(member):
        if member.mtime >= datetime:
            return member
        return  ignored
    return notolderthan

@Action
def prettyprint(val):
    print val

if __name__ == '__main__':
    listdir('.') >> notolderthan('2/3/08') >> prettyprint

The module cmdlet just contains the Cmdlet and Action class definitions as found in the orginal web posting.

Wing IDE

So I have finally found programmer editor nirvana, suprisingly it is a closed source program! A long time ago I was trying to put together a vim IDE, I blogged about my early trials, but never finished describing my setup, which in the end was quite nice. But seeing a lab mates matlab setup made me super jealous for a modern debugger, with tool tip variable value pop ups! Some searching around and I rediscovered Wing. Being a better programmer, and understanding what a full featured debugger can do for me now, there was no going back. I just purchased the educational version today, and it is the best 100 dollars I have spent in memory. The company is super cool, answers question quickly, and the product is to die for. If you program in python check it out!

It has been a while . . .

Well I have been trying to do my own web programming at mudskipper.ca but I just never seem to finish anything. So I loaded up wordpress, but then I was onset by an oppressive amount of comment spam. And shortly after gave up. So without further ado I am back at wordpress.com, which upon further reflection is simply awesome.

A lot has happened. Some highlights giving my first talk in vancouver, going to the SciPy08 conference as a supported student, as well as having a beautiful new cat named Ada.

Stickley Joy!

I have dreamed of this joystick, and now it lives . . .

Vim IDE update

Well the weekend was a whirlwind tour of vim’s features and customizations. This was largely enlightening to me, I truly love vim, and this experience has only deepened that feeling.

Some customizations of note:

• The NERD Commenter, for smart programming comments.
• python.vim, updated highlight script, adds some pretty bells and whistles.
• pydoc.vim, give access to pydoc output in a split vim window. I have defined the extra keymap (nmap K <leader>pw) which is more natural in python mode than having access to shell man pages.
• taglist, a great class browser.
• python.vim, updated indent rules that are more inline with the python recommendations (align with open braces, etc).

These are the major enhancements I have used. They solve some of the editing and informational problems outlined in a good IDE. But it is not perfect yet, and will need some tweaking. the pydoc interface is not super smart, as it is confused by imported modules. Omnicompletion is very slow, and works poorly — though I am considering using the short cut <C-Space> as the completion instead of <C-x><C-o> as I find the latter awkward. I am still not sure about using smart tabs — that is not allowing tabs past allowed limits. I have written some code to enforce this, but it might be overly draconian.

I have also set up pylint as the python compiler, so I get error messages that I can correct according to style guidelines. This is awesome, though I need it to open another window so I can see the errors in a more explicit manner.

Anyway, the progress continues. I will have to put my head around the symbolic debugger. There seems to be a good project for gdb, called clewn, though it will be seen how easy it is to use pdb instead.

Vim as a python IDE

I have been programming python in much of my spare time, largely inspired by http://www.swc.scipy.org/. As a result I have started to use subversion, and have become increasingly interested in software engineering. I now write unit tests for much of my code, and have felt a tremendous increase in quality and enjoyment in my everyday coding!

Recently I also tried using WingIDE which is an amazing product. I wish it was more straitforward to define a keyboard alias (such as Control-[ short for Escape). I can only figure out how to use keyboard macro’s for built in functions — which is more than I need . . .

All this brings me to the topic of this post, I want WingIDE like functionality in vim. This is mostly motivated by poverty, I find the Personal addition of Wing to be missing must have features (source assistant, and svn integration), and Professional is way out of my budget. This made me think more about vim, which is supposedly highly customizable, people even argue it is already ready to be a modern IDE. I find said post a little short on details, and I simply am not will to use the command line debugger after having tasted the sweetness of a proper integrated debugging experience.

With this in mind I am now ready to collect the plugins and know-how to recreate the features of Wing that I find amazing. The top ones that come to mind are:

• Symbolic debugging!
• Subversion integration.
• Source assistant (this shows the documentation for each function, object, variable, etc . . . it is really nice).
• Smart completion using tab (I will want to use Ctrl-n, Ctrl-p to navigate the pop-up, which was something I could figure out how to do in Wing). This means that I will have to figure out how to get vim to only tab when it makes sense (that is if I press tab twice, it will do nothing, as indentation is only legal for block structure). I like this secondary feature greatly, I remember similar functionality from emacs, so it should be doable.

I am sure there are others that I will notice, but if I have this I will be very, very happy. This might mean I will have to learn to script vim with python, which will be interesting itself. I will write about this as I make progress. Ultimately it will be nice to collect plugins, tips etc into one package that is documented and easy to install that gives some IDE feeling to vim (and not just say it is possible, which is what I mostly see).

Note: I have seen pida, and I find it poorly documented, and outside of subversion integration it covers IDE features I care less about. That being said, it might serve as a basis for the work I envision. Though I worry it solves a more general problem than I am interested in. An IDE platform seems overkill to me, I would rather have vim, and only vim give the features.

Desert xterm

I have been using the gnome terminal since my recent move to ubuntu (lts). Sadly I find it to be slow (scrolling in vim, etc can lead to choppy performance on my crappy computer), so I have moved back to the oh-so-lovely xterm.

Being a die hard fan of the built in gvim colorscheme desert, I felt it was time to update my .Xdefaults. It was a bit trial and error to figure out which color* resources did what, but once this was done, using the gimp and the color picker it was speedy.

Here is a screenshot of vim in the terminal, editing some fortran code:

For those interested, here is the .Xdefaults

!
!XTerm*underLine: on
XTerm*colorMode: on
XTerm*colorBD: lightyellow
!XTerm*colorBDMode: off
XTerm*colorUL: yellow
XTerm*colorULMode: on
XTerm*dynamicColors: off
XTerm*highlightSelection: true

!———————————————————-
! Colorscheme based on the gvim colorscheme desert
!———————————————————-
XTerm*background: #313031
XTerm*foreground: White

!colors
!(unknown)
XTerm*color0: #000000
XTerm*color7: #bebebe
XTerm*color8: #666666
XTerm*color9: #cf6171
XTerm*color10: #c5f779
XTerm*color11: #fff796
XTerm*color12: #4186be
XTerm*color13: #cf9ebe
XTerm*color14: #71bebe
XTerm*color15: #ffffff

!(1 is an option color in .Xdefaults)
XTerm*color1: #ffa2a5

!(2 is a resource color in .Xdefaults)
XTerm*color2: #bdb66b

!(3 is for def/return color in python)
XTerm*color3: #f7e78c

!(4 is for comments color in python)
XTerm*color4: #84cfef

!(5 is for import/from color in python)
XTerm*color5: #ce5d5a

!(6 is for function color in python)
XTerm*color6: #87ff87

!set the font, and fond size
! this also uses a antialias/scalable font
xterm*faceName: Andale Mono
xterm*faceSize: 12

!set the window geometry
XTerm*geometry: 110×40

Dynamical Systems, and Continuation

Well it has been a while since I have been writing simulations for food web models. More often then not I have resorted to writing specialty programs in fortran. Though I knew of AUTO, I found it to be unbelievably cryptic to use, as a result I forged ahead.
Well I am back at looking for a better way. programs I have found, and deemed to be the only actively developed projects are:

1. AUTO 2000
2. PyDSTool
3. Matcont
4. xppaut

I am interested in finding out which will be the best suited to my needs. So far I really like the idea and design of PyDSTool, making it a programming language, especially in python, is quite nice. Sadly until they have support for continuing periodic solutions, it is not a stand in solution for me. AUTO also has a python scripting support, but it retains the cryptic use of 2-3 letter parameters, which are meaningless to me, seconds after I write them. I also find it diffucult to use the results I get from AUTO. It feels like a stand alone tool.

Xppaut on the other hand is tremendously easy to use, blindingly fast, but hard to decrypt the save files. Though I feel I will continue to use it in place of AUTO, until PyDSTool develops further. To this end I will create a python module that will parse the "write all information" save file from the AUTO session of xppaut. Hopefully this well serve the duel purpose of making me a better python programmer, so that I can begin to add to PyDSTool, as well as getting a working solution for the short run.

Finally I guess I need to look into matcont, which seems to be a very mature, and powerfull solution, strongly related to PyDSTool. The only reservation I have is that I would need to use matlab, which being commericial, and expensive, makes me wary.

Well, I need to start somewhere

So, I have a blog. I plan to get some working ramblings on LaTeX, Fortran 95/2003, Postscript like graphics, and some of the research I am thinking about.

Lets see how it goes.