My Name Rhymes

DNA on Programming

2008-02-16T17:45:47Z

"Well, what we called a computer in 1977 was really a kind of electric abacus, but..."
"Oh, now, don't underestimate the abacus," said Reg. "In skilled hands it's a very sophisticated calculating device. Furthermore it requires no power, can be made with any materials you have to hand, and never goes bing in the middle of an important piece of work."
"So an electric one would be particularly pointless," said Richard.
"True enough," conceded Reg.
"There really wasn't a lot this machine could do that you couldn't do yourself in half the time with a lot less trouble," said Richard, "but it was, on the other hand, very good at being a slow and dim-witted pupil."
Reg looked at him quizzically.
"I had no idea they were in such short supply," he said. "I could hit a dozen of them with a bread roll from where I'm sitting."
"I'm sure. But look at it this way. What really is the point of trying to teach anything to anybody?"
This question seemed to provoke a murmur of sympathetic approval from up and down the table.
Richard continued, "What I mean is that if you really want to understand something, the best way is to try and explain it to someone else. That forces you to sort it out in your own mind. And the more slow and dim-witted your pupil, the more you have to break things down into more and more simple ideas. And that's really the essence of programming. By the time you've sorted out a complicated idea into little steps that even a stupid machine can deal with, you've certainly learned something about it yourself. The teacher usually learns more than the pupil. Isn't that true?"
"It would be hard to learn much less than my pupils," came a low growl from somewhere on the table, "without undergoing a pre-frontal lobotomy."
"So I used to spend days struggling to write essays on this 16K machine that would have taken a couple of hours on a typewriter, but what was fascinating to me was the process of trying to explain to the machine what it was I wanted it to do. I virtually wrote my own word processor in BASIC. A simple search and replace routine would take about three hours."
"I forget, did you ever get any essays done at all?"
"Well, not as such. No actual essays, but the reasons why not were absolutely fascinating. For instance, I discovered that..."
He broke off, laughing at himself.

-DNA

Iterating Over Database Results in C#

2007-12-17T01:13:00Z

At my job, we use what is essentially a custom C# ORM. Soon after I arrived, we adopted this idiom to pull a DB connection from the pool and run a query:

 class Something
{
    public int ourIdiom(long identifier)
    {
        using (DBConnection db = new DBConnection())
        {
            using (IDataReader rec = db.execQuery(@"
                SELECT some, rows
                FROM  Table1 t1, Table2 t2
                WHERE t1.t2id = t2.id
                AND   t1.id = ?",
                identifier))
            {
                while (rec.Read())
                    doSomething();
                return 1;
            }
        }
    }
}

Which is nice because it disposes of both the connection and the dataReader properly on failure, but ugly because it uses a whole bunch of boilerplate. Unfortunately, because doSomething() needs to execute inside both using statements, and we weren't using C# 2.0 until a few months ago, the only way to avoid the boilerplate would have been to pass in delegates.

This would have been just as ugly as keeping the boilerplate in the code, so we stuck with the using² idiom.

Since our switchover to 2.0 a few months ago, I've had an idea that I could make this idiom more concise and remove some boilerplate that I didn't get a chance to try until this week. Using 2.0's iterators, which seem awfully familiar from my python work, that can be reduced to:

class Something
{
    public int ourIdiom(long identifier)
    {
        foreach (IDataReader rec in new SQL().Query(@"
            SELECT some, rows
            FROM  Table1 t1, Table2 t2
            WHERE t1.t2id = t2.id
            AND   t1.id = ?",
            identifier))
        {
            doSomething();
        }
        return 1;
    }
}

With some fairly simple code:

public class SQL : 
    System.Collections.Generic.IEnumerable<IDataReader>
    {
        string m_sql;
        ArrayList m_sqlparams;

        public SQL()
        {
        }

        public SQL query(string sql, params object[] sqlparams)
        {
            m_sql = sql;
            m_sqlparams = new ArrayList(sqlparams);
            return this;
        }

        IEnumerator<IDataReader> IEnumerable<IDataReader>.GetEnumerator()
        {
            using (clsDBConnection db = new clsDBConnection(m_dbname))
            {
                using (IDataReader rec = db.execQuery(m_sql, m_sqlparams))
                {
                    while (rec.Read())
                        yield return rec;
                }
            }
        }

        //this is required because IEnumerable<> inherits IEnumerable. bleh.
        IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator() }
    }
}

This code should come in handy for more reasons than just its cleanliness. First off, if you ever need to change the idiom for accessing the DB, it's stored conveniently in one place.

Second, It makes a prepared SQL call into something like a thunk: a bit of code representing a future computation that can then be passed around to be performed later. An example:

void setUpQueries(nameId, addressId, custId)
{
    List<SQL> stringsWeNeed = new List(new object[]{
        new SQL().query("SELECT name FROM names WHERE id=?", nameId),
        new SQL().query("SELECT addy FROM addresses WHERE id=?", addressId),
        new SQL().query("SELECT zip FROM customers WHERE id=?", custId)});

    List<string> strings = getOurStrings(stringsWeNeed);

    mangleStrings(strings);
}

List<string> getOurStrings(List<SQL> queries)
{
    List<string> strings = new List<string>();
    foreach (SQL query in queries)
    {
        foreach (IDataReader rec in query)
            strings.Add(rec.GetString(0));
    }
    return strings;
}

While the example is silly, this property can be exploited to seperate declaration from execution, which I often find to be a useful pattern.

Though I am far from the first person to discover this simplification, I did figure it out on my own and I thought it was neat enough to share.

A quick disclaimer: the code here is modified from the original to remove some details, and is completely untested, and probably doesn't compile. You may, however, use it or distribute it as you see fit; it's licensed under the wtfpl.

UPDATE: added the while(rec.Read()), which I'd forgotten

Why Applications Suck (Part 1)

2007-08-24T01:50:00Z

Why do people outside of corporations overwhelmingly use webmail instead of email clients? Long before rich email clients like gmail were available, people have flocked to the internet email client - messaging has been the killer feature of the web for a long time now.

But why?

The answer is simple: Overwhelmingly, users do not want to install programs, and users do not want their data stored on their computer. In that order of importance.

Users Don't Want to Install Programs

Let's examine the process of getting a popular, free, email client like Thunderbird installed. To do so, imagine that we're instructing our kindly aunt Tilly how to install it.

Now Tilly is no old fogie, she's been using Microsoft Word at work for over a decade now, and she knows her way around msn.com. However, she just decided to strike out looking for a new job, and wants to finally use that personal email address that came with her cable internet. She's got us trapped on the phone, and "we know computers", so here's roughly what we'll tell her to do:

type google.com in the address bar
type in "thunderbird"; press enter
click on the first result
click on "Download Thunderbird"
click "OK"
choose a place to save it (Just use the desktop, Tilly!)
minimize the browser
hunt for the file named "Thunderbird"

And, ohmigod, we're not even close to done. She's still got to run the installer (3-4 clicks minimum), which involves selecting an installation directory (a what?), then open the program, open the tools -> accounts menu option, find the documentation for her service provider's email tools, copy them into the "add new account" wizard (another 3-4 clicks) where appropriate, remember her user name and password, and then download all the email she's had sitting there since she hasn't checked it ever.

Now, finally, she's ready to use her email client. It's just that easy!

Let's compare the process for getting her up and running with gmail:

type "gmail.com" in the address bar
click on "sign up for gmail"
fill out a form (a task she probably knows how to do)
click on "I accept. Create my account."

Now, Tilly may not be the most technological mind around, but she's also no dummy. I know which one I'd rather direct her to do.

Desktop application install is a problem.

Do We Even Want to Fix This?

Many web 2.0-savvy types will tell us that this isn't a big deal. In the future, all of our applications should be inside our browser, and to speak otherwise is heresy! The desktop is dead! Sure, Javascript + Json + the browser has limitations, but those restrictions are liberating! Get used to it, or perish!

Well, bah humbug. Here's what's good about the web: The easy posting, viewing, and navigating of documents described by URIs.

That's it.

Let me repeat it for clarity: The web, in your browser, is only really good at sending discrete data, displaying discrete documents and navigating in between them.

Phew.

The whole paradigm is absolutely superb at getting this job done. The system of hyperlinks, URIs, DNS, and HTTP over TCP/IP is the greatest achievement that computer programmers have achieved, a massive global effort that has managed to stay amazingly open and coherent. It "Just Works" enough to change the way information is distributed, businesses are run and even how society is structured.

Anything amenable to being represented as discrete visual documents has taken to the internet like a duck to water. Chief among them is email; check out the top 100 global Alexa websites and see how many "social networks" in how many languages are in the top 100. Sites displaying news (documents), photographs (documents), videos (strictly as documents), and information about software (documents) make up most of the rest.

What's left among those 100 is the elephant in the room; the Killer Internet Application: search. Even Microsoft's lame search engine finds its way into the top 100 because people will do whatever they need to in order to search the web; it's the only way to get an effective handle on the information in the Great Hyperlinked Mass. I believe it's the exception to my "web is only good at documents" theory that proves the rule.

Search Works, so Let's Put All our Apps On the Web

Since the web gained massive worldwide popularity in the mid-1990s, an enormous amount of effort has been spent on making the browser into an application development platform. The Great Mozilla Rewrite basically shut that browser down between 1998 and 2002 in order to make it an application platform. Microsoft developed activeX, Sun pushed Java applets as the solution, Adobe developed SVG, and Macromedia developed its Flash player.

Nowadays, Firefox is pushing <canvas>, Microsoft has released Silverlight, Macromedia has become Adobe, and Flash is the only one of the first generation technologies still breathing oxygen as a development platform.

Furthermore, with the invention of Google Gears, Joyent Slingshot, the aforementioned Silverlight, or Adobe Air, you might soon be able to (gasp) save files to the host computer! That's right, in 10 years we've almost gotten to the point where a program running in a browser can save a file on your hard drive and read it when you're offline.

The problem is that, fundamentally, a browser is still only good at one thing: documents. Its whole paradigm is set up that way. What does the "back button" mean when you're playing a game of asteroids? What does "home" mean while you're editing a photo? Does an address bar on an mp3 player need to be so in your face?

We Can Fix This

So, instead of shoehorning our applications inside of the browser window because it's convenient, let's do something radical: bring our "real software" applications into the modern age of the internet.

In the next article, we'll talk about how software installation came to suck so badly, with an eye towards how to go about bringing it up to the web era.

MySQL Partial Dump

2007-08-17T00:05:00Z

At work, I'm trying to bring our testing from its current ad-hoc state into something more useful. Priority #1 at the moment is making what limited test suite we have into a practical, repeatable test of our main code library.

To do so, however, I need to create script to import a useful subset of the data from our nine gigabyte MySQL production database into a local database. Unfortunately, mysqldump seems not to support the export of only N rows from a database table, and neither does there seem to be a standard way to do so. In fact, there seems to be no standard way to automatically generate insert queries, so it's not even possible to hack it with OUTFILE tricks.

Indeed, a quick look look at the mysqldump source code shows that it just constructs a query of the form: "SELECT * FROM table [WHERE condition] [ORDER BY field]":

dynstr_append_checked(&query_string, 
    "SELECT /*!40001 SQL_NO_CACHE */ * FROM ");
dynstr_append_checked(&query_string, result_table);

if (where)
{
  /* snip */
  dynstr_append_checked(&query_string, " WHERE ");
  dynstr_append_checked(&query_string, where);
}
if (order_by)
{
  /* snip */
  dynstr_append_checked(&query_string, " ORDER BY ");
  dynstr_append_checked(&query_string, order_by);
}

And then pieces together an insert string manually:

while ((row= mysql_fetch_row(res)))
{
/* snip */
  for (i= 0; i < mysql_num_fields(res); i++)
  {
    /* snip */
    if (row[i])
    {
      if (!IS_NUM_FIELD(field))
      {
        /* snip */
        unescape(md_result_file, row[i], length);
        /* snip */
      }
      else
      {
        /* change any strings ("inf", "-inf", "nan") into NULL */
        char *ptr= row[i];
        /* snip */
        else if (my_isalpha(charset_info, *ptr) ||
                 (*ptr == '-' && my_isalpha(charset_info, ptr[1])))
          fputs("NULL", md_result_file);
        else if (field->type == MYSQL_TYPE_DECIMAL)
        {
          /* add " signs around */
          fputc('\'', md_result_file);
          fputs(ptr, md_result_file);
          fputc('\'', md_result_file);
        }
        else
          fputs(ptr, md_result_file);
      }
    }
  }
}

So, rather than deal with hacking the rather baroque C (in a proprietary source control that I've never used, no less) to add another option to limit output rows, I threw together a little python function that uses MySQLdb to do what I want:

"""
get_n_rows retrieves n rows from a given table on MySQLdb connection conn

conn:  open MySQLdb connection object
table: string; name of the table which we are generating queries for
n:     int; number of rows to output from table

Legal kwargs:
    reverse: bool; whether to return rows in reverse order (requires idCol)
    idCol:   string; results will be sorted on this column if reverse is 
             True
    where:   string; provides the "where" clause of the select query
             NOTE: escape this text yourself! use connection.escape() .

Released under WTFPL (http://sam.zoy.org/wtfpl/) - use this code as you 
wish. Note that it's not tested at all, and certainly won't handle blob 
fields. That said, it worked for my purposes today.

bill.mill@gmail.com 8/17/07
"""
def get_n_rows(conn, table, n, **kwargs):
    cur = conn.cursor()

    #TODO: copy mysqldump's table escaping function
    sql = "select * from `%s` " % (table)
    if "where" in kwargs:
        sql += "where %s " % (kwargs["where"])
    if "reverse" in kwargs and kwargs["reverse"]:
        sql += "order by `%s` desc " % (kwargs["idCol"])
    sql += "limit %s;"
    cur.execute(sql, (n, ))

    i = 0
    row = cur.fetchone()
    insert_stmts = [] 
    while row and i < n:
        i += 1
        #note that the MySQLdb source suggests that conn.literal is 
        # private; I see no problem with using it, as it just loads the 
        # default escaping functions. Caveat Emptor.
        strow = ", ".join([conn.literal(o) for o in row])
        insert_stmts.append("(%s)" % (strow))
        row = cur.fetchone()

    cur.close()
    
    return """
-------- INSERT DATA FOR TABLE %(table)s -----------
    
    LOCK TABLES `%(table)s` WRITE;
    DELETE FROM `%(table)s`;
    INSERT INTO `%(table)s` VALUES %(sql)s;
    UNLOCK TABLES;

-------- END INSERT DATA FOR TABLE %(table)s -----------
""" % {"table": table, "sql": ",".join(insert_stmts)}

On a side note, the MySQLdb docs are somewhat hard to track down; I found them after some effort here.

Why I Like My Macbook Pro

2007-05-13T13:12:00Z

Summed up very succinctly:

This means that my Macbook Pro, which I received on December 26, 2006, has been up and running, without a reboot, for 36% of its active life.

This is despite the fact that it's been my alarm clock every morning during that span, so it's gone to sleep and woken up at least once every single day, and usually more than once. Furthermore, it's been used for development in C, Python, erlang, C#, IronPython, Pypy, and probably a few others. I've installed many programs during this time, viewed many PDFs, probably opened 10,000 tabs in Firefox, Safari, Opera, and Camino, and run quite a few Windows sessions with Visual Studio in Parallels.

This laptop stability is a wholly new experience for me; Windows and Linux both have required fairly frequent reboots on the laptops I've had, and I've had several between 2000 and now. It's awesome to be able to trust the sleep and hibernate modes of the computer; certainly it's foreign to this Linux convert.

Now, that doesn't mean I've turned into a raving Mac fanboy. I think much of their software sucks, the dock still annoys me, and their computer is nigh on unusable without Quicksilver. However, their laptop hardware and its integration into the OS has been all that I hoped for and more.

UPDATE: A few hours after posting this entry, a security update forced a reboot :)