Steve Reads

Category: Technology

A SQL question! — July 11, 2012

A SQL question!

/stevereads/2 Comments

Suppose you have

SELECT a+b AS some_sum, a-b AS some_diff
FROM some_table

It would be syntactically invalid, as far as I understand it, to have

SELECT a+b AS some_sum, a-b AS some_diff,
some_sum + some_diff AS something_else
FROM some_table

, because `some_sum` and `some_diff` aren’t names that yet exist in the scope of that `SELECT` statement. To use `some_sum` and `some_diff`, you need an outer scope that has access to all the names created in the inner scope — so you need either

SELECT some_sum + some_diff AS something_else
FROM
(
SELECT a+b AS some_sum, a-b AS some_diff
FROM some_table
)

or, if you don’t need to attach a name to `some_sum + some_diff`, you can just do

SELECT a+b AS some_sum, a-b AS some_diff
FROM some_table
HAVING some_sum + some_diff >= 0

(or whatever it is that you might want to do with `some_sum + some_diff`)

So … are there some nightmare scenarios that would result from allowing the names ‘`some_sum`’ and ‘`some_diff`’ to be visible in the inner scope? Does anyone know the SQL spec well enough to understand the problem?

__P.S.__: Oh, I think I can answer the question a moment after asking it: the problem is that you can create a name in the inner scope that’s the same as the name of a field in the base table. That is, you can do something like

SELECT 2*a AS a, 3*a AS triple_a
FROM some_table

, and that `3*a` statement is ambiguous: are you referring to triple the *derived* `a`, or triple the *base* `a`?

Even still, I can imagine allowing one scope to refer to a name created within that scope, but then making it a syntax error to introduce a name in a given scope that’s the same as a name in a base table. The way to resolve the ambiguity in this case would be to do

SELECT 2*a AS double_a, 3*(double_a) AS sextuple_a
FROM some_table

I wonder if a Sufficiently Smart Compiler (“a classic argument often pulled out in a LanguagePissingMatch”) could find all ambiguous uses of names, or whether there’s some logical reason why it just couldn’t.

Then again, forbidding the creation of a name in a given scope that is the same as the name of a name in an inner scope or base table would make a lot of common usage impossible. I routinely do

SELECT one_thing, sum(something) something_else
FROM some_table
GROUP BY one_thing

I.e., I’m reusing `something_else` in this scope, even though it also exists in the base table. If I had to constantly use new names for derived fields, just because the derived fields might be referred to in the outer scope, I think I’d find that annoying.

Then, of course, there’s the ever-available argument that it’s just simpler to implement a compiler which imposes certain mild aesthetic constraints on its users.

Tools for making the impossible possible — January 29, 2012

Tools for making the impossible possible

/stevereads/Leave a comment

I’ve been thinking a lot lately about tools that help make the difficult easy, which has got me thinking again about probably my favorite quote of all time, by A.N. Whitehead:

> It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilisation advances by extending the number of operations we can perform without thinking about them.

I think about this at work all the time, because our use of SQL makes possible a lot of data-gathering and -analysis tasks which really would have been impossible without it. Answering some question about many thousands of servers (100,770 as of the end of September, 2011) would be unimaginable. Certainly getting many *quick* answers to many *quick* queries would be absolutely unimaginable. Without some tool that allows quick aggregation across many different datasets, we’d have to resort to home-brewed scripts that, say, ssh to thousands of machines in parallel and ask them questions. Or we’d have to reinvent Query, more likely.

There are two aspects to SQL that I think about constantly at work: first this trick of turning an impossible problem into a triviality, and second the sense of *playfulness* that it enables. It takes only a tiny bit more effort to turn from the question you were trying to answer into something unexpected, something more general, or something more nuanced. Often answering questions that you didn’t know you had involves finding a table you didn’t know the company published, which in turn involves asking around to see who would know best about a given kind of data. Finding answers to questions you didn’t know you had seems to me part and parcel of what SQL is all about.

The term “generative technology” gets at this. I would link to the first Google search result for this, except that I don’t really like how Jonathan Zittrain — who is, in fairness, most associated with this term — runs with it. iPhones versus non-iPhones isn’t really at all related to what I have in mind here, and I don’t think the definition he has there gets at what even he means by it. The term “generative” comes ultimately from generative grammar, which my non-linguistically-trained self understands to mean “a set of simple rules for the formation of sentences, which rules can be combined in infinitely many ways to construct infinitely many distinct sentences.” In mathematics, think of axioms and rules for their combination: there aren’t that many axioms defining the integers, but they can be combined with only a few more rules (about ordered pairs and what, exactly, multiplication of two ordered pairs means) to build rational numbers, and thence real numbers, and thence complex numbers. The simple axioms, and simple rules for their combination, lead to infinitely complex objects.

(Because I cannot resist a filthy quote when given the opportunity, it’s here that I’ll quote Stephen King’s advice on writing: ‘When asked, “How do you write?” I invariably answer, “One word at a time,” and the answer is invariably dismissed. But that is all it is. It sounds too simple to be true, but consider the Great Wall of China, if you will: one stone at a time, man. That’s all. One stone at a time. But I’ve read you can see that motherfucker from space without a telescope.’)

And so it is with SQL and other generative technologies. They don’t give you a single product that you use in the 10 or 20 or 100 ways that you’ve been told to use it; in this sense, I view Facebook as non-generative. A generative technology, like Unix or SQL, might have a steep learning curve, but once you’ve learned it you can do infinitely many things.

There are lots of complexities once you’ve learned the atoms, and even once you’ve learned how to combine the atoms. In Unix, for instance, your first task is learning to string together programs with pipes. Once you’ve done that, you’ll soon enough be writing your own programs. but you have to write them in the Unix Way, which often involves allowing them to sit with a pipe on their left side and a pipe on their right; in this way, they themselves become part of the generative toolkit. Again, invoking Whitehead, the point is to make complicated action reflexive and doable without thinking. Take a common Unix pattern:

[some commands] | sort | uniq | sort -nr

This takes the output of [some commands] — assumed to contain one interesting pattern per line — and displays it in descending order of frequency, with the frequency in the left column and the pattern on the right. This isn’t many characters, so typing it out becomes second nature; the smart thing to do, though, would be to put this Unix fragment in its own script, which we might call sort_by_pop (or really ‘popsort’, which would save you some keystrokes: there’s already a command that starts with ‘sort’, but no commands that start with ‘pops’, so ‘popsort’ would be easier to get from tab-completion; Unix people think this way):

(19:44 -0500) slaniel@example.com~$ cat sort_by_pop
#!/bin/bash
sort |uniq -c |sort -nr

Now you can just pipe things through sort_by_pop if you want to sort them by popularity:

(19:44 -0500) slaniel@example.com:~$ grep -o ‘^[^ ]+’ access.log |sort_by_pop |head
607 46.165.197.141
520 173.242.125.206
309 199.21.99.67
229 130.195.253.1
169 66.249.71.18
162 72.14.199.102
83 72.246.0.10
37 142.167.21.94
34 198.228.223.217
33 80.58.205.47

Hm, what’s that grep(1) bit? Looks like that bit of script could be usefully abstracted into something called ‘get_ip’:

(19:48 -0500) slaniel@example.com:~$ cat get_ip
#!/bin/bash
grep -o ‘^[^ ]+’

whence we simplify to “cat access.log | get_ip | sort_by_pop”. Now you don’t need to understand the nuances of how sort(1) and uniq(1) work if you don’t want to; in fact, you may never need to know that those atomic tools are sitting underneath your molecular abstractions. If you trust the person who wrote the tools, you can assume that get_ip gets an IP address from a suitably formatted Apache access log, and that sort_by_pop sorts a file containing one pattern per line in descending order of popularity.

And so forth. The idea is to constantly combine the atoms of your knowledge into larger and larger molecules, which allows you to forget about the individual atoms unless you really need them. (Where you often need to remember the atoms is for performance reasons.)

In SQL, one way of combining atoms into higher-order molecules is by means of views. A view is a new table (“relation” for the relational-calculus purists in the room) constructed from lower-order “base tables”. There may be some very clever way to get by without views, but I don’t know what it might be. Often you’ll end up with a query that requires you to join one complicated sub-query to itself; without views, you’d be repeating the sub-query, which would probably involve copying and pasting a bunch of text. This would make editing one of the sub-queries a hassle, because you’d have to repeat your edits once for every sub-query. With views, you create the view once, give it some shorthand name, then use the shorthand on every subsequent reference. Any edit only has to happen once, in the view. Again, the point is to make higher-order thought effortless.

(Java, by contrast, requires so much boilerplate that it gets in the way of quickly scanning a piece of code and understanding what it’s trying to do. Either that, or it requires the developer to carefully shunt his boilerplate off into a little boilerplate area of his code. Or it requires the code reader to develop a finely honed skill of skipping over boilerplate. One organizing principle for writing code of any sort ought to be that it puts the least possible distance between the task you’re envisioning and the code you write for it.)

Having developed such a love for SQL, and having long ago learned how to build high-order castles in Unix, I’m now on the hunt for other generative technologies that will make difficult tasks possible. My goal for 2012 is to discover such a set of technologies for time series. It’s not just a matter of writing formulas that allow me to manipulate time series in any way I see fit, though that’s hard enough (it will probably involve R, and may also involve [book: Data Analysis with Open Source Tools], recommended in the highest terms by my awesome friend Dan Milstein). And it’s not just a matter of manipulating them in a way that makes exploring them, combining them, and being surprised possible, though that’s part and parcel of the generative idea.

Rather, the difficulty with making these things work right starts, it seems to me, way down in the guts. Akamai’s Query system is brilliant — one of the most brilliant technologies I’ve ever seen at a company, central to everything I do at every minute of every day — and works so well because there’s a lot of stuff going on under the hood which, again, I mostly don’t need to think about. The low levels do break, just as they do in any software system (all abstractions are leaky); and when they break, I’m forcibly reminded that all my simplifying abstractions rest very tentatively on a lot of lower-level foundations. Without someone doing a lot of low-level grunt work, Whitehead’s dictum doesn’t hold. (Perhaps the grandest abstractions of all in the modern world are “the market economy” and “industrial democracy” — abstractions that we forget are based on very concrete things like cheap fossil fuels or policemen who will enforce contracts at the point of a gun.) In the case of SQL, someone has to build a backend data-storage method that allows quick lookups. In the case of time series, what will the backend storage system look like? Will we need something like MapReduce? Do we need a different high-level language to concisely encapsulate high-level time-series concepts like “the trend component” or “the spectrum”?

Here is the place to note a lesson that I find I have to repeat to myself over and over: don’t think any harder than you need to. My interest in time series is very non-abstract; I have some specific questions I want to answer about some specific datasets at work. And yes, I want to make sure that I can combine them in new and interesting ways in a reasonable amount of time. But until I’ve asked a single specific question of a single specific dataset, I shouldn’t think too hard about making an apple pie from scratch.

So anyway, there’s a general point in here, and a specific one. The general point is to hunt for abstractions that make it possible to get a lot done without thinking, and make it possible to explore areas you didn’t even know you *could* explore. The specific point is that in 2012, I want to see what I can do with Akamai’s time-series data. I imagine one of these points will be interesting to you, the other less so.

I still need someone to explain to me what problem Google Plus solves, and why it’s not creating other problems that I find way more annoying — July 19, 2011

I still need someone to explain to me what problem Google Plus solves, and why it’s not creating other problems that I find way more annoying

/stevereads/8 Comments

I am now on Google+, because men of my ilk are required to join new services such as this. But it is not solving a problem I have. It’s just creating more problems.

I left Twitter some months ago, because my workflow was like this:

1. post to Twitter.
2. have the post automatically mirrored to Facebook.
3. flip over to Facebook to make sure that the post replicated.
4. if it took more than a minute to replicate, refresh a few times before giving up with an odd, low-level form of nervousness.

I was connected to all the same people on Twitter that I was connected to on Facebook. Granted, there were others on Twitter who were not on Facebook, like famous people or, to put it another way, people with whom the interaction was expected to be more one-way. I followed Chris Onstad, author of Achewood, for instance. It was fun. His Twitter feed is hilarious. I followed the author of a book I really loved, and to my great surprise and pleasure he connected me with the [newspaper: Boston Globe] to write a piece for them. So I can definitely say that Twitter was good for my wallet.

But it was also really distracting, for reasons I laid out in that piece. And I don’t have the self-control to be connected to two social networks and yet only check them occasionally. Nor do I have the self-control to prevent myself from refreshing email dozens of times a day. Email by now is in fact a reflex. But at least email isn’t getting new updates constantly. And the fun of Twitter is following lots of interesting people saying lots of interesting things. It was *too* fun, honestly. Sad to say. It’s taken me until age 33 to realize that I actually have to talk myself down from unhealthy habits; I literally have to say to myself, “Yes, you want a milkshake from Toscanini’s made with burnt-caramel ice cream and a shot of espresso. But 1) those are empty calories, 2) you can use that $5 for something better, 3) you’re trying to cut down on your caffeine intake, aren’t you?” I probably should have been doing this throughout my life. In any case, I’m starting now, and it seems to be working.

So I ended up thinking that it was better to concentrate on one social network, namely Facebook, and ditch the rest. Really, it’s probably best to go with zero social networks, but Facebook is essentially inevitable now. Many websites require you to use a single-sign-on platform, and OpenID is dead; Facebook is the only one left. Google, as I recall, tried OpenID. OpenID sucked. So a Facebook account is … well, “necessary” isn’t the word, but it would certainly be annoying to go without one.

Also, though, Google+ isn’t solving a problem that I have. Its main innovation, so far as I can tell, is that it makes the concept of a “circle” fundamental: circles for friends, circles for acquaintances, circles for colleagues, etc. Others have the problem that they need their messages to go to specific circles; I do not. The problem is supposed to be that, say, you want to write naughty words, but don’t want your grandmother to read them; or that you want to bust on your coworkers but don’t want your coworkers to read it; or some such. First of all, Facebook offers enough in this direction that Google+ is not really solving a real problem. For instance, I was connected on Facebook to my girlfriend’s son, and I set up permissions such that he couldn’t see a lot of what I posted. Problem solved. I’ve certainly been on the receiving end of those blocks, too: coworkers have clearly put me on a “don’t share your Wall with coworkers” rule set. Facebook also offers a nice preview feature, which allows you to see how your profile looks when visited by a specific Friendster of yours.

More fundamentally, having to figure out who should read a given post and who shouldn’t is the kind of psychic weight that I try very hard to cast off. “Should this go to Acquaintances, or only to Friends?” is a question which, when asked often enough, will eventually rub my mind raw.

I’d much prefer to just post whatever comes to my mind to Facebook or Twitter, and let people decide whether they want to read it. On the receiving side, I’m sure Facebook has enough intelligent algorithms to decide whether you find my posts interesting; if it doesn’t think you do, they won’t show up in your feed. If you think I swear too much, you can ignore my posts. If you really hate me, you can defriend me.

Mark Zuckerberg, I’m told, said at one point that people should have just one persona, which is on display at all times. At some level this is wrong: I make dirty jokes with some people but not with my mother; I have one side of my personality that really likes discussing policy and theory, but that side will quickly put itself into hiding if it senses that the people around it just don’t care about those things.

But the costs of having different personalities on the web — one for Acquaintances, one for Friends, one for Policy Wonks — exceed the gains, for me anyway. I’d much prefer to put all those personalities together into one feed. I suspect the audience finds this more interesting, in any case: much better to get the occasional notional pornography title (a category of Facebook post that I very much enjoy writing) amidst links about health insurance, I think, than to get a steady diet of one or the other. I could be wrong about this, in which case my audience would prefer that I put different ideas in different channels. Not my problem. I guess Google+ is designed for people who think that *is* their problem.

Google+, I gather, came out of watching people’s occasional paroxysms of anger against Facebook’s privacy problems. But people don’t really care about their privacy; if they did, Twitter wouldn’t be popular. In Twitter, everything you write is visible to the world. Had Facebook decreed from early on that everything you post is public, people would have nothing to be angry about. It’s the perception of a bait-and-switch that angers people about Facebook; it is manifestly *not* that people care about their privacy. So inasmuch as Google+ gives people more privacy than Facebook, it’s not offering the world a solution to any problem that the world actually suffers from.

Neither Google nor Facebook actually cares about privacy. What they want is to sell ads, or to otherwise monetize their social networks. That means, as the saying goes, that “if you’re not paying for a product, *you’re* the product.” They’re selling *you* to advertisers. So at a fundamental level — the level of what keeps the lights on in Google’s and Facebook’s datacenters — neither of them is actually interested in your privacy. Neither of them *could* be interested in your privacy. If Google+ has any actual appeal, it’s the perception that it won’t bait-and-switch you. But that’s just responding to an accident of Facebook’s history. If Facebook were born now, it would be Twitter, and everything would be public. Not baiting-and-switching on privacy is not the basis for a network that people should care about.

What I get with Google+, then, is the solution to a privacy non-problem and the creation of groups of acquaintances that cause me stress without solving any problem I have. Can someone point me to some really killer feature that I should know about?

I am often a curmudgeon about new technology, but it’s not out of reflexive hostility toward new things. It’s that I really need people to prove to me why I need something. I was this way with the iPhone and the Mac, but eventually the tipping point came where it was obvious that the iPhone and the Mac were just better than every one of their competitors, and that there was no legitimate reason to avoid buying one. So I’m more than open to being convinced that I should use Google+. I just need to be convinced that it a) solves a problem I have and b) is better than Facebook. Thus far I’ve not been convinced.

A realization about writing and programming — March 29, 2011

A realization about writing and programming

/stevereads/5 Comments

Level-zero programmers obsess over essentially unimportant details of programming: whether you should write blocks like

void someFunction(void) {
printf “Yesn”;
}

or like

void someFunction(void)
{
printf “Yesn”;
}

, for instance, or whether to use vim or emacs to edit your code. These are unimportant for at least a couple reasons. First, you can solve these sorts of style problems in an entirely automated way. Second, there are many, many things that are more important than these sort of nits; they don’t even really impact your code’s readability, for one. These details don’t even count as ‘style’. They’re purely mechanical.

A level-zero English-language writer pays attention to Strunk & White. What’s odd about Strunk & White is that, far from being about [book: The Elements of Style] like its name suggests, it’s really [book: The Elements of Grammar]. True, grammar does matter. Inasmuch as “style” means “writing to appeal to your audience,” and inasmuch as your audience cares about little grammar nits, grammar is important. Grammar, in this view, is arbitrary but important. It’s like the location of the silverware at dinner, or like not chewing with your mouth full: it’s an arbitrary convention that certain groups of people pay attention to. It’s a class identifier. Using “whom” properly, or not ending a sentence with a preposition, is a class identifier. It’s a way to signal to people of your class that you’re one of them. Others will completely miss the signal, which doesn’t make them rubes; it just means that they don’t subscribe to your particular class signals. So call it [book: Elements of Writing For The Wealthy], perhaps.

But even with that caveat out of the way, and even if you don’t agree with Language Log that Strunk & White is a pile of trash, it’s still the case that [book: The Elements of Style] is wildly, comically unimportant to the act of writing readable text. Whether you use “whom” properly (and I’m a dues-paying member of the Whom-Using Board Of Pedants) has practically nothing to do with whether people will read your writing all the way through to the end.

Grammar is important to get you off the ground, in a certain sense. Works riddled with typos are often hard to get through. Using commas where you mean to use semicolons will sound wrong in your reader’s ear *if he’s trained to read them that way* (so again, the rule “know your audience” is logically prior to the rule “use punctuation properly”).

But that’s just the point: this is level-zero stuff. These are the rules you pay attention to because they’re rote and mechanical, and thereby easier to remember and implement than “grab your reader with a good hook” or “use lots of examples when you’re arguing an abstract point.” They’re high-school rules; they’re not adult rules.

Some probably obvious observations on economics, inspired by Apple, which just suggest that I need to read more economics — March 5, 2011

Some probably obvious observations on economics, inspired by Apple, which just suggest that I need to read more economics

/stevereads/Leave a comment

(__Attention conservation notice:__ 1400 words thinking aloud about innovation, Apple-style, and what connection it might have to the standard, boring sort of competition that you read about in introductory economics.)

I’ve become somewhat obsessed with Apple in recent months (see “The iPhone is a gateway Apple product”). They’re an easy company to get obsessed over, because they build the best products. When Google was building the best products, like their search engine or the maps app, I was obsessed with them too. Most of their other products are quite good, but they’re not *perfect* in the way that the iPhone is, in the way that the Google search app is, or in the way that Google Maps is. Every time I use Google Calendar — and I use it, mind you, a dozen times a day — I’m reminded of all the things it could do better. I never think that about Google search or about the iPhone. They are perfect.

It’s been remarkable to watch Apple’s competitors. Apple invented the iPod 10 years ago, and it has owned the category ever since. Others have tried to compete with them, but haven’t managed to produce anything even comparably good. Likewise with the iPhone. When I remember what preceded the iPhone, my mind is kind of blown. Pre-iPhone phones were thought to be such poor computers that manufacturers decided to invent their own alternate universe, including their own poor substitute for HTML, rather than just *put a fast computer in your pocket*. Other companies have had four years to respond, including at least one company that makes a lot of money and should, by all rights, have beaten Apple at this game.

Yet they’ve not. Not even close. Android continues to be the technology of tomorrow, just as Linux has always been, and one strains not to say that it will *always will be* the technology of tomorrow.

I see in this the simplified picture of markets that we read about in introductory economics. Someone starts a company — say, a bakery making artisanal bread in a big city. There’s unmet demand for this bread, so people flock to it. The lines run out the door, and the bakery is habitually sold out by noon. They ramp up their production and add some machines to augment human labor. Maybe they raise their prices. Now the lines are shorter (translation: prices have risen, so quantity demanded has decreased), but the bakery’s total amount of income has increased (translation: price elasticity is less than 1). Now the bakery is making lots of profit.

Other bakeries see this profit, and they want in. So they move into the market and try to do the same thing more cheaply. Maybe doing it more cheaply is harder, because the incumbent bakery makes its artisanal bread using giant machines that can produce individual loaves for not very much money at all (translation: high fixed cost to buy the machines, low marginal cost per loaf). Anyone who wants to move into the market would either have to make better bread for the same or higher price (think of Starbucks moving into a world of Maxwell House), or make equal-quality bread more cheaply. (Translation: high fixed costs are a barrier to entry, and make monopolies more likely.)

But if the incumbent baker makes money consistently for years, it may eventually happen that banks take notice and loan someone the money to start a competing bakery at scale. (Imagine here AMD moving into a world dominated by Intel.) Now the incumbent has to lower its prices. In some perfect-competition models, every last customer flocks to the cheaper bakery. The bakeries alternate investing huge amounts of capital to produce at cheaper prices at larger scale. We’re in a price war.

This may be a good thing — we get cheaper bread — but it’s not what Apple is doing. Instead, they’re *innovating*. They’re not running down the slippery slope of a price war. Instead, they’re making products that no one had and no one knew they wanted before. (I *still* don’t really need an iPad, though it would be handy for reading academic PDFs.) If they were a different kind of company, they could instead try to make cheaper widgets than their competitors, but where’s the joy in that? If that’s how companies operated, we wouldn’t even have BlackBerrys by now; we’d just have cheaper little knock-off LG or Nokia “feature phones.”

There are *some* models I remember learning in college that tried to capture this. They usually fall under the labels ‘imperfect competition’ or ‘monopolistic competition’. Coca-Cola has a legally enforced monopoly on the term ‘Coca-Cola’, for instance. But that doesn’t really capture innovation. Most of the models I remember learning, and nearly anything captured under the term ‘perfect competition’, don’t describe the sort of market where companies innovate.

I still need to finish reading Schumpeter’s [book: Capitalism, Socialism, and Democracy], but he touches on a similar idea in there. His famous term from that book is “creative destruction”: capitalism’s great contribution to the world is that it constantly destroys industries and replaces them with new ones. No one laments all the horse-and-buggy drivers put out of business by the automobile. There are legitimate reasons to lament the end of businesses built around physical newspapers, and a just society will try to help laid-off newspaper workers land on their feet. But the innovation of the web, which gave rise to an entirely new industry that caught incumbents unawares, makes life better in many well-known ways.

One of Schumpeter’s main arguments (the book contains many arguments and ranges far afield; it argues, for instance, that soon capitalism will be destroyed when its large corporations turn into mere offices for the filling out of forms in triplicate) is that capitalist enterprises shouldn’t mainly fear that a new competitor will produce the same product for cheaper, but rather that an entire industry will come into being that renders the incumbents’ whole reason for being moot.

There’s an “innovation through cheapness” argument along these lines, most famously laid out in [book: The Innovator’s Dilemma]. It goes like this: there’s some incumbent that makes a big, expensive product that’s the cream of the crop and whose lead seems impregnable. Think of Oracle databases, for instance, or the Sabre airline-reservation system, or Microsoft Windows. Some cheap competitor comes along, producing a product that doesn’t do most of what the big guys do, but does it for much less money (MySQL databases, Internet airfare searching, Linux). Initially, the big guy couldn’t care less about the newcomer — might not even notice the newcomer, in fact. The newcomer may have in fact taken away the big guy’s most annoying customers: those customers who care mostly about price, or those who demand a lot of features for not a lot of money. Good riddance, says the big guy.

Now that the newcomer has some customers, it can start adding features. Because the big guy has been around for a while, competitors have had a while to look at what customers are buying. The newcomer can bring fresh eyes to an existing market, too: the big guy has established sales forces that are trained at selling a specific kind of product, has a large bureaucracy that’s specialized in making their current product lines, and is slow to move in response to change. The newcomer is small and can be more agile. They keep adding features and taking away increasingly important customers from the big guy.

Eventually the big guy takes notice, but by this point it’s too late: the world has shifted entirely to the fresh, cheap product that the new guy is making. This hasn’t happened yet with Windows, of course. Oracle bought MySQL. A quick scan suggests only modest declines in travel-agent employment over the next 7 years. I should try to think of some other examples.

That’s at least two distinct types of innovation: innovating through inventing entirely new product lines that render existing products moot, and innovating through producing simplified subsets of existing products. The latter seems different than merely making a lower-cost version of the same product; that’s classic price competition.

Now, I’m pretty sure I never heard any coverage of innovation when I took economics classes in college. Is there any good modeling of this sort of thing?

Agile — January 31, 2011

Agile

/stevereads/Leave a comment

I promised on Facebook that I’d say some things about the Agile software-development idea. I’m by no means an Agile expert, but it made a lot of sense to me. There were a few components that I really liked when we used it at my last job. They’re pretty dyed-in-the-wool Agilists; I probably would be if I really dove into it, but as it is I’m picking and choosing the bits I liked. Here goes:

* __Daily stand-up meetings.__ You and your team literally stay standing long enough for each of you to explain a) what you did yesterday, b) what you’re going to do today, and c) what’s holding you back, if anything. If you spend too many days in a row saying, “I meant to finish x, but I didn’t, and I swear it’ll be done today,” eventually those on your team are going to ask if they can help you get past whatever roadblock you’re encountering. Which is a polite way of saying, “Come on, now; hurry it up.”

Staying standing is vital here. Meetings get over a lot more quickly if you’re on your feet rather than kicking them up and sipping a cup of coffee.

* __Feelings of human shame__. I feel like the last bullet calls out one important part of Agile: it’s built on the very deep-seated need not to fail in front of others. It’s certainly possible in a lot of organizations to hide in a corner and fail for a long time before anyone notices. Agile makes that much less likely: if you fail, you’ll fail within a few days in front of your whole team. Better to ask for help than flounder pridefully.

* __Monthly kick-off__: You tell everyone what you’re going to do. They get a chance to ask you why you chose to prioritize one thing over another.

* __Monthly show-and-tell__: You and your team stand up in front of everyone in your organization (sensing a pattern?) and show what you did; they get to evaluate, among other things, whether you did what you said you would do during the kickoff. Preferably everyone on the team does this for his or her own part of the project. Again, you do not want to be the person who failed to get your thing done.

* __Small teams with a few defined roles__. In our case the roles were developers (the people actually writing the code), QA engineers (testing), and “product owner.” The product owner is the voice of the customer. She (and I’m happy to say that this isn’t just political correctness overcompensating: we really did have a delightful, brilliant, energetic product owner who happened to be a woman, along with plenty of kick-ass male product owners) tells the developers what we need to build and lets us figure out how to build it. Then she talks to customers and figures out what to have us do. And she tries as hard as she can to clear shit out of our way so that we can get work done. I was a part of at least one meeting filled with three or four engineers, that our product owner interrupted halfway through to say, “Do the engineers really have to be here for this?” whereupon, the answer having been in the negative, we got up and left. See “__Few meetings, fast, standing up__,” below.

* __Short iterations__: The interval between planning and show-and-tell is pretty critical. You don’t want it to be too long: saying, “This project will take a year to complete” is as good as saying that it’ll never be done. But you don’t want a show-and-tell every week; that’s too much bureaucratic overhead — or maybe not, if that’s what your team decides is right. But it certainly feels to me like there’s a tradeoff: keep going until the marginal benefit of shortening deadlines is just offset by the marginal cost of larding the process up with meetings.

There are several important reasons to have these short cycles. One is, as I mentioned, that planning for a small task is (duh) easier than planning for a long one; you’re much more likely to end up accomplishing what you said you would. Second is that you’re *outrunning your customers*, namely those inside and outside your company who want you to do things for them. The faster you can get something rough and ready in front of them, the less likely they are to say, “You know what would be really nice? …” or “Oh, sorry, turns out that we fired the CEO and now we’re looking for something different.” You’ve also given them something to anchor their expectations: now instead of comparing what’s in their head to a blank canvas, they’re comparing it to something real. That’s likely to focus people and ground them.

Certainly the most exhilarating work experience I’ve had since leaving college was building a rough prototype at my last job. We got something out the door in, if memory serves, 3-4 weeks (might’ve even been more like 2) and in front of customers. We asked them right away: is this something you want? Yes, the UI is crappy, but ignore that; we’ll make it look nice later. (Those of you objecting here that design — of software, of posters, of books, of whatever — is fundamentally bound up with the functioning of the product and isn’t just about “making it pretty”: yes, you’re right, but I’m pretty damn convinced that rough-and-ready is a tradeoff worth making.) Does it do, functionally, something you’d want? What else would you want it to do? Does this replace something you use in your everyday work? After you get done using this tool, what’s the very next thing you’d go do?

Once we had those answers, and we were convinced that real people would really use this thing, we sat head down at our computers for a couple weeks (I think we were on two-week cycles here) and got another block of interesting work done. People came by during the show-and-tells, looked at it, critiqued it, and told our product owner what else it needed to have. Heads back down. After a few months of this, I looked up and — hot damn! — we had a really nice piece of work in front of us. I can’t say I was responsible for big chunks of the code, but I did what I could, and I felt proud of what we’d accomplished.

* __Few meetings, fast, standing up__: This isn’t part of according-to-Hoyle Agile, and you could have it as part of any organizational structure, but I suspect that organizations which take Agile to heart will want to use this too. At my last job we took Paul Graham’s notion of maker time versus manager time to heart. The idea there is that, for a manager, the hourlong meeting is the quantum of work; the manager’s day is scheduled in one-hour blocks. Whereas writing software — and maybe this is true of other focused endeavours — takes a few hours to get into. Me, I needed the music to be set up right, I needed maybe a cup of coffee, I needed to knock off some smaller programming challenges to get the muscles limbered up … it could take me an hour or two just to get going. An hourlong meeting thrown into the middle of that bungs everything up: by the time you get out of the meeting, you forget where you left off and need to do the coffee, the programming exercise and so forth all over again.

Meetings are good for some things. If you’re hashing out a strategy, or the structure for a few weeks of work, you probably need meetings. But to the extent that you can shunt them onto email, or onto quick in-person chats in the hallway, you’re creating space for your makers to make stuff. (By the way, I was extremely cautious — maybe too cautious — about approaching people, for exactly this reason. First I emailed them, which seemed like the least-intrusive way to get in touch. Then I IMed them. Then I walked over to them and chatted. Yes, the last method is likely to be fastest *for me*, but it’s also likely to break their concentration.)

* __Test-driven development__. This is even less a part of classic Agile, and if you’re a certain sort of person maybe you feel you don’t need it. Me, I needed it. The idea here is that you write the tests for your code first, then write the code, then confirm that your code passes the tests, then write some more tests, then write some more code. At every iteration, write a failing test just as fast as possible. For me, this was a way to combat my own inertia, so this is sort of like __Feelings of human shame__: Agile fundamentally seems like a way of adapting an organizational structure to fit human behavior. When I forced myself to write a failing test as fast as I could, I was fighting my own habit of overthinking a problem before I started solving it. Yes, thinking is important, particularly when it comes to design, but there’s a certain kind of thinking that is either a) telling you why a certain thing won’t work, or b) pre-emptively making a solution elegant before you’ve made it at all. Test-driven development (TDD) says that elegance can wait just a bit. Write your test, then write your solution, then confirm it works (test passes), then make it elegant (i.e., more general, i.e., refactored). You’ll get the elegance soon enough, but in the meantime you’ll have working code. And who knows: by skipping the elegance for a bit, maybe you’ll find that a whole different approach works better.

Like I said, this isn’t really by-the-book Agile. But these are all things that make a lot of sense to me, and worked great for us.

Little iPhone UI details — October 12, 2010

Little iPhone UI details

/stevereads/1 Comment

(__Attention conservation notice__: 800 words documenting the near-perfection of the iPhone user interface. Plus a small suggestion for how they could improve it still more.)

Things the iPhone does that I’ve not seen anyone else do:

* There’s a ‘.com’ button when you’re typing in a field that accepts domain names (like an email-address field, for instance). I only realized recently that you can press and hold the .com button to get .net, .edu, .org, and .us.

* It gets cooler. Add an international keyboard (Settings -> General -> Keyboard -> International Keyboards -> Add New Keyboard…), then go back to an URL field (like in Safari, say). Suppose you added an Arabic keyboard. Now look at what the .com button has: top-level domains for Arabic countries, like .ae. Similar things happen if you add French keyboards, etc.

* Spell-check will not flag words if those words are in your address book. It does better than just not flagging them, actually: if you type a friend’s name in lowercase, it’ll correct the case for you.

* Probably most every iPhone/Touch user has noticed by now that you can go into the Maps application and start typing the name of someone in your contacts for whom you have a physical address; the Maps app will offer you any matching physical addresses.

You’d expect — or at least, *I’d* expect — Google to get this right, too. After all, when you’re signed into your Google account, Google knows about your contacts; it should be easy enough to carry that contact information over into Google Maps. But they don’t.

* This next one is easier to describe by example. I have a friend, Chris Rugen, whom I’ve jokingly put into my iPhone contacts as “Chris Rügen”; my iPhone contacts sync with Google. In the iPhone, if I start typing “rug”, it offers up “Rügen” as a completion — even though the “u” that I typed has no umlaut over it.

I’ve found no other system that does the completion this intelligently. Thunderbird doesn’t. Google itself doesn’t, either: searching within my Gmail contacts for “rugen” doesn’t return the accented contact. In either Thunderbird or Google, I need to start typing Chris’s actual email address — which contains no accent, of course — in order for them to find Chris.

These are all just little things, but that’s *exactly* what makes the iPhone what it is: nearly all the little things are done perfectly. You get a sense of calm when you play with an iPhone (and “play,” by the way, is exactly the right verb), because nothing is out of line with what it should be. Computers have a habit of steadily accumulating frustrations; the iPhone does not.

One thing the iPhone does need to do differently is related to the Archive button in the email client, which only arrived in one of the new iOS releases (I want to say 4.0). If you have a Gmail account, the Archive button will do the same thing on the phone that the Archive button does on Gmail’s website. That’s great. But there’s no Archive operation available for non-Gmail accounts. Worse, the Archive button gets replaced with a delete button for non-Gmail accounts. So if you’re used to archiving messages by tapping the leftmost button, muscle memory alone will often make you delete messages accidentally. This gets especially to be a problem now that iOS 4 does a single combined universal inbox: you don’t know which account a given message is coming from (could be Gmail, could be not), so the very same inbox view can sometimes make that button do archival, sometimes do deletion. It’s a dangerous combination. (Though not too dangerous: you can always retrieve the deleted message from the Trash, if you notice soon enough that you deleted rather than archived.)

What’s odd is that fixing this to work with all account types wouldn’t be that hard, unless I’m missing something. Right now you can configure which folders on your remote server will be used for drafts, sent mail, and deleted messages: go to Settings -> Mail, Contacts, Calendars -> [account name] -> Account Info -> Advanced and look under Mailbox Behaviors. If the folders you’ve specified don’t exist, I believe the iPhone mail client will create them. It would be easy enough to add an ‘Archives Mailbox’ item under there.

I can see a reason why they might not do this. Gmail’s archives folder is called ‘All Mail’ on the server side, so a sensible default name for the archives folder would be ‘All Mail’. But on a non-Gmail account — IMAP, say, or Exchange — maybe that name would be confusing. Maybe on those sorts of accounts, it would be smarter to call the folder ‘Archives’. But then you’ve got an inconsistency between the name of the archives folder on different types of server; that may confuse users.

Unified messaging on the iPhone (or anywhere, really) — September 22, 2010

Unified messaging on the iPhone (or anywhere, really)

/stevereads/1 Comment

Know what would be really handy? To include all your friends’ and acquaintances’ contact info — including Twitter handles, Facebook profiles, phone numbers, email addresses, RSS feeds, [foreign: und so weiter] in some global address book (like the nice Gmail contacts list, which you can sync with a mobile device, and which I sync quite happily with the iPhone), then gather together all those items and make them on the device. Quite often a conversation starts via text message, moves over to email, maybe ends up in a voice chat, turns into a blog post, etc. Wouldn’t it be nice if the device could record your voice chats, or at least transcribe them?

Barring that, just being able to search text messages at the same time as you search emails would be a big win. Didn’t BitPim do this?

How Google’s PageRank algorithm works — August 23, 2010

How Google’s PageRank algorithm works

/stevereads/Leave a comment

There was recently a little tempest in a teapot on Twitter about some measure of how influential a Twitterer you are, which got me thinking about how silly a lot of these measures are. They remind me a lot of early baseball statistics: read one of the early books in the sabermetrics genre, like [book: The Hidden Game Of Baseball], and you’ll find a lot of dudes with limited statistical training taking a little of x, and a little of y, adding it together, and finding that — hey, this looks like a good measure of something; let’s use it! There are worse ways to develop a measure of something, but there are also better ways. Subsequent sabermetricians developed better ways, using rigorous statistics.

These amateur Twitter ranks made me think of Google’s PageRank algorithm, which was based on years of work trying to measure influence in academic journals. The Google problem and the journal problem are quite similar: if a given journal, or a given journal article, have a lot of inbound “links” (that is, citations from other journals), that’s good — but only if the linking journals are themselves highly linked. A bunch of citations from garbage journals, or a bunch of inbound web links from spam sites, shouldn’t contribute anything to your journal’s or your website’s rank.

The math behind the basic PageRank algorithm is simple. Imagine that you’re an infinitely patient web surfer who jumps randomly from page to page: you start on some page, click on every link on that page, and do the same on every subsequent page. If you click on 10,000 links, and 500 of them go to CNN.com, then you’ve spent 5% of your time on CNN. If 1,000 of those links go to NYTimes.com, you’ve spent 10% of your time on the [newspaper: Times]. The PageRank for a given site is the fraction of time that our hypothetical random web surfer spends, in the long term, on that site.

Specifically, imagine a big table listing the probability that one site links
to another:

CNN NYTimes HuffingtonPost TalkingPointsMemo
CNN 0.75 0 0 0.25
NYT 0 0.75 0.05 0.20
HuffPo 0.35 0.40 0.20 0.05
TPM 0.20 0.30 0.10 0.40

(In Google’s practice, this would be a square table with as many rows and as many columns as there are pages on the Internet. So imagine a table with billions of rows and as many columns.)

The probability in each cell is the probability that the domain in the same row links to the domain in the same column; so for instance,
the probability that CNN links to Talking Points Memo is 0.25; it doesn’t link at all to the [newspaper: Times] or to the Huffington Post.
(These are purely made-up numbers.)

Imagine, again, that we’re a random web-surfer. We start at the [newspaper: Times]’s website, click somewhere, then click another link at random from where we’ve landed. So now we’re two clicks away from wherever we started. What’s the probability that we’d end up at Huffington Post after two clicks? Turns out that we can answer the question if we turn the table above into a mathematical object called a matrix. The matrix version of the table above looks like

[ [0.75 0 0 0.25] [0 0.75 0.05 0.2] [0.35 0.4 0.2 0.05] [0.2 0.3 0.1 0.4] ]

This is called a “transition-probability matrix.” Since every row sums to one — because every page links to some other page, even if the page it links to is itself — it’s also called a “stochastic matrix.” It turns out — via the Chapman-Kolmogorov equation — that the probability of getting from one of those pages to another within two clicks is the square of the transition-probability matrix. Squaring such an odd-looking beast may seem weird, but it’s a well-defined operation called matrix multiplication. The square of that matrix looks like so:

The square of the matrix shown above

which says that the probability of getting from Talking Points Memo to Huffington Post in two clicks is 0.075. Continue this process through many clicks and you’ll eventually approach what’s called the “stationary distribution” — the long-term probabilities of ending up in any given state, independently of where you started out. In this particular case, the stationary probabilities are

* CNN: about 0.285
* The Times: 0.4
* HuffPo: about 0.057
* TPM: about 0.257

The [newspaper: Times], then, would have the highest PageRank, followed by CNN, TPM, and HuffPo. Again, the interpretation of the PageRank is very simple: the long-term fraction of time that a random [1] web-surfer would spend on your page.

Of course Google needs to modify this basic approach somewhat. One assumes that there’s a lot of “secret sauce” baked into the actual PageRank algorithm — as opposed to this bare skeleton — that allows Google to respond to spammers more effectively.

Regardless, I suspect a Markov-chain approach like PageRank would apply with few modifications to Twitter. Your Twitter rank would go up the more retweets (someone on Twitter essentially forwarding your tweet on to his or her followers) or mentions (someone including your Twitter handle in his or her tweet) you get, but only if the people retweeting or mentioning you were themselves highly ranked. The essential metric would be the fraction of time that a random Twitter surfer would spend reading your tweets.

One might argue that having a lot of followers should increase your Twitter rank. This is debatable: are your followers actually valuable if they’re not mentioning you or voting for your tweets by retweeting them? I could see basing the Twitter rank on followers rather than tweets, in which case the interpretation would be “the fraction of time a random Twitter surfer, jumping from Twitter handle to Twitter handle, would spend on your account.” But that seems less promising than basing it on tweets.

In any case, the point is that PageRank has a simple interpretation based on probabilities. This is in contrast to all the gimmicky Twitter-ranking sites, whose point is to drive traffic to one site or another. It’s like we’re reinventing Google. Right now we’re at the Alta Vista of Twitter: the current competitors are less focused on searching and ranking, more focused on their “portal,” and ultimately less professional than what Google came up with by using actual math.

For much, much more on the mathematical details of PageRank, see Langville and Meyer’s [book: Google’s PageRank and Beyond: The Science of Search Engine Rankings]. It’s a gem.

[1] — Specifically, someone surfing according to a Markov chain. This is a very simple way of modeling a sort of random process that has “no memory”: if you tell me the first, second, third, fourth, and fifth pages you clicked on, and ask me to guess what your sixth page will be, I can throw out all the information you gave me apart from the fifth page. Earlier states, that is, don’t matter in prediction of the future; only the most recent state matters.

An axiom for life in the era of Twitter and Facebook — August 11, 2010