How should scientific codes be constructed? Scientific codes are often experimental in nature, and therefore need to be easy to change, and, of course, quick to develop in the first place—who wants to invest any more than necessary into an idea that might not work out after all? On the other hand, computational scientists are an impatient bunch: Once the method stabilizes, or it becomes clear that it is generally useful, more and more demands get placed on a prototype code, most of all speed. It is clearly desirable to respond to these demands in an evolutionary manner. Why rewrite an entire code, possibly in a different language, if only one small part is responsible for most of the run time?

One answer that I have found to address these problems well is what I call hybrid development. Nearly all my code these days is “hybrid”. Now what does that mean? That means, I start developing in a high-level language that is quick to modify and separates me from the more menial tasks of coding, allowing me to code and test an algorithm quickly. The cost for this quick development turnaround is that the initial prototype is often fairly slow because many small operations are performed in the (often interpreted, instead of compiled) high-level language. This is an effect that everybody who has developed larger-scale code in MATLAB or similar environments has seen. Often, the bulk of the run time is concentrated in 10% or less of the entire code. In this case, the sensible solution is to replace the slow part of the high-level code with a lower-level compiled implementation, while keeping as much of the existing high-level code intact. This is the idea of hybrid development.

MATLAB supports this idea to some extent with its MEX mechanism. There are however many more powerful tools on the market (and available for free) that make hybrid development easier and faster by orders of magnitude. In a future installment of this mini-series, I will highlight the software environment I use for my hybrid codes, and point out what I believe its advantages over other solutions are.

This is a distribution of ARPACK, the well-known sparse eigensolver, with a number of changes:

• the most recent patches from the ARPACK download page applied
• a nasty, potentially crashing bug fixed
• the build system replaced by autoconf/automake

Note that ARPACK is not my own work, I am only redistributing a modified version. I contacted Rich Lehoucq years ago with a report of the crash bug, but the fix never showed up on the ARPACK page, so I resorted to distributing my own modified ARPACK package. Since ARPACK continues to be useful for my work, but its rather old-fashioned build system kept getting in my way, I one day decided to replace it with a modernized, autotoolized version, so that it can be built on most machines with just the regular

./configure
make
make install

incantation. Additionally, the new build system makes it trivial to build a shared library of ARPACK. Since there doesn’t seem to be a proper version-numbering scheme at ARPACK’s original site, I’ve also taken the liberty of starting at version 0.90.

The current source tree can be found in my git repository, and downloaded from here. You may also obtain a copy directly by running the command

git clone http://git.tiker.net/trees/arpack-autotools.git

So after madman is currently orphaned in Debian and might get kicked out of the archive along with xmms kicking the bucket, I just noticed that one of my newer pieces of code made it into Debian: tagpy! :) That’s awesome.

To everybody who is holding their breath waiting for a madman update: I still use madman for my own music managing needs, as it still (surprise) is exactly what I want in a music manager. But I simply don’t have enough time for serious maintenance. I might switch it to audacious in the near future to keep it working for me, but I’m not promising extensive new features…

All source control is now handled through http://git.tiker.net. Briefly: To obtain the most recent copy of some piece of software, type

git clone http://git.tiker.net/trees/NAME.git

MyPyrecious

A perty Python clone of Bejeweled. At present, I’m a bit uncertain about whether I may redistribute the sound bits which I used in the game. They’re just scavenged from the internet. Until I figure something out, only the parts of the game which are mine are available. Currently, the only way to get this game is to download the git tree.

SokoSolve

A solver for Sokoban levels. See this post for details. To be found here.

I’ve uploaded FemPy to my arch repository for your perusal. A release is coming soon.

This page (and the information on it) is obsolete. Do not use the Arch trees any more—they will go away soon. Please head over to my source control page for the latest and greatest.

This page lists my arch trees, along with their contents. To use these trees, get a copy of baz and type

baz register-archive http://tiker.net/archives/*year*-public

After that, you can get the newest version of the software by saying

baz get inform@REMOVETHIStiker.net--*year*-public/*category*--*branch*--*version*

Note that clicking the links in your browser gives you a 403 Forbidden HTTP error. This is intentional. baz can access these trees just fine.

2006-public

This repository can be found at http://tiker.net/archives/2006-public. It currently contains this:

  fempy
    fempy--mainline
      fempy--mainline--1.0
        base-0

  fempy-suite
    fempy-suite--mainline
      fempy-suite--mainline--1.0
        base-0

  pyangle
    pyangle--mainline
      pyangle--mainline--1.0
        base-0

  pylinear
    pylinear--mainline
      pylinear--mainline--1.0
        base-0 .. patch-1

  pymbolic
    pymbolic--mainline
      pymbolic--mainline--1.0
        base-0

  pytools
    pytools--mainline
      pytools--mainline--1.0
        base-0

  tagpy
    tagpy--mainline
      tagpy--mainline--0
        base-0 .. patch-1

Stuff will branch over from the 2005 tree as time goes by and changes are made.

2005-public

This repository can be found at http://tiker.net/archives/2005-public. It currently contains this:

  aufschrieb
    aufschrieb--mainline
      aufschrieb--mainline--0.0
        base-0 .. patch-2

  clockwork
    clockwork--mainline
      clockwork--mainline--0.0
        base-0 .. patch-2

    clockwork--releases
      clockwork--releases--1.0
        base-0

  fempy
    fempy--mainline
      fempy--mainline--1.0
        base-0 .. patch-1

  find-the-word
    find-the-word--mainline
      find-the-word--mainline--1.0
        base-0

  ixlib
    ixlib--mainline
      ixlib--mainline--1.0
        base-0 .. patch-3

  madman
    madman--production
      madman--production--1.0
        base-0 .. patch-37

  ocaml-library
    ocaml-library--mainline
      ocaml-library--mainline--0.0
        base-0

  pyangle
    pyangle--mainline
      pyangle--mainline--1.0
        base-0

  pylinear
    pylinear--mainline
      pylinear--mainline--1.0
        base-0 .. patch-8

  sdlucid
    sdlucid--releases
      sdlucid--releases--0.91
        base-0 .. patch-1

  tagpy
    tagpy--mainline
      tagpy--mainline--0
        base-0 .. patch-7

    tagpy--releases
      tagpy--releases--0.90
        base-0

  tvinfo-ripper
    tvinfo-ripper--mainline
      tvinfo-ripper--mainline--0.0
        base-0

  wgwam
    wgwam--mainline
      wgwam--mainline--1.0
        base-0

This repository is now closed.

2004-public

This repository can be found at http://tiker.net/archives/2004-public. It currently contains the following trees:

  madman
    madman--production
      madman--production--1.0
        base-0 .. patch-24

  mide
    mide--mainline
      mide--mainline--1.0
        base-0

  pyangle
    pyangle--mainline
      pyangle--mainline--1.0
        base-0

  pylinear
    pylinear--mainline
      pylinear--mainline--1.0
        base-0 .. patch-1

  pyrecious
    pyrecious--mainline
      pyrecious--mainline--1.0
        base-0 .. patch-1

  sokoban-solver
    sokoban-solver--mainline
      sokoban-solver--mainline--1.0
        base-0 .. patch-7

  wgwam
    wgwam--mainline
      wgwam--mainline--1.0
        base-0 .. patch-5

This repository is now closed.

I’ve been molding this site according to my publishing needs, and in the process of doing so, I’ve ventured into the Drupal source code. I’ve made a few changes that I believe are generally useful. Besides submitting them to the respective maintainers, I’ll post them here for added accessibility.

• I’ve modified the Drupal core to provide “Clean URLs” without needing proper mod_rewrite support. Instead, you only need to be allowed set the ErrorDocument to the provided 404.php page. There is a slight complication that the implementation handles properly: HTML forms submitted via HTTP POST cannot use these clean URLs since the form data does not propagate through the ErrorDocument redirection. This NOT FOUND: drupal-4.5.1-clean404-2005-01-04-v3.patch=patch against version 4.5.1 of Drupal implements this. (upstream)

  Update 1: It turns out that my web hosting company, Alturo, does support this. In order to enable it, you need to say RewriteBase / in .htaccess. They do not document that, though. (In general, they document fairly little.) As a result of this, I won’t be maintaining the Clean-URLs-via-404 patch.

  Update 2: I’ve put a modified version of the patch onto the Drupal bug tracker that applies to current Drupal CVS.

• I’ve modified and fixed inline.module in several ways in this NOT FOUND: inline-2005-01-04-v1.patch=patch: (upstream)

  • You can now specify [ inline:filename.ext] instead of only the number, for added link stability.
  • A bold, red NOT FOUND message is shown if the given file name or number does not exist.
  • The module previously generated spaces around the img and a tags. Those are gone.
  • Images are only inlined if linked to with [ inline:...]. If you say [ file:my.png], this will remain a file link.
  • The recent fix for multiple links in one node is part of this patch.
  • (The spaces were necessary to keep inline.module from picking up these example links.)

• I’ve added two patches to the title module.

mide is a minimal IDE born out of the need to automate the build-test-edit cycle. Its user interface should be pretty self-explanatory: Enter a make command, hit f9, wait, double click on an error message, correct the error, repeat. To compile it, you need Qt version 3 along with the uic. (Qt 2 should work, I haven’t tested that. If you do, please let me know.)

A simple “make” should be enough for building, you might have to adjust the Makefile a bit (this works fine for my debian system). Installation is even easier: Copy “mide” and “mide-backend” to your path, that’s it. mide-backend is a script that takes care of invoking the correct editor backends. It receives the following arguments:

mide-backend save <i>backend</i>
mide-backend open <i>backend</i> </i>file</i>
mide-backend position <i>backend</i> <i>file</i> <i>line</i>

Currently, only the NEdit backend works, and even for this to work, you need a patch to nedit which is included in the distribution. If you implement more editors, please send me the appropriate snippets of mide-backend. mide will have some zombie processes in most cases, which will go away when mide dies, this is a design limitation in Qt that’s hard to work around, sorry.

Hey, and look: It’s even version 1.1 now. Version 1.1 adds correct handling of warnings and tooltips containing the error on examine.

Download

You may download the latest version from my software repository.

The current arch tree is called mide--mainline--1.0.

So, I was wasting time before going to bed last night, like, by setting up this site, and by playing Sokoban. KSokoboan to be precise. Now, I made it up to level 12 of David W. Skinner’s Microban level set, which is rightfully said to be pretty easy. Well, level 12 wasn’t, at least not to me. (I admit that that’s embarassing, but who cares.)

So, I made it to the “I can prove this unsolvable” stage of puzzle solving. But since I reckoned that someone would have noticed that this level is unsolvable, my mind immediately went to cheating. Unfortunately, nobody had published move sequences to solve Microban. I was absolutely clueless as to how this level might be solvable, and I just had to know. So, I wrote a Python program to solve Sokoban levels. You’ll find sokosolve.py and a few demo levels in the sokoban-solver module of my 2004-public arch tree.

The program itself has two nested applications of A*, a usable (but half-baked) heuristic and a complete game model. In just 215 lines. Python never fails to impress me. Attached to this story you can see a dump of the program solving the level that had me so confused.