Stuck on a Hill

In artificial neural network (ANN) research, there's a well known problem of local minima (or maxima). I've worked a bit with ANNs but much more with a (superior) learning algorithm, Support Vector Machines (SVMs). Unlike the latter, ANNs require heuristics to "converge" on an optimal solution given some very large decision surface. The heuristics, to simplify a bit, are intended to get the algorithm to converge on a global, not local, solution. Local solutions are unwanted because they can appear to be global (good) solutions, given some snapshot of the decision surface, but in fact are very bad solutions when one "zooms out" to see the larger picture. This phenomenon is perhaps best illustrated with geographical imagery. If I am walking up and down hills, en route to a very large mountain, a local maxima might be the top of a foothill. But it would hardly be a global maxima, like the top of the mountain. The point is that ANNs can converge on the foothills, telling us that it's the mountain.

ANNs notoriously suffer from this limitation, but the "blindness problem" is endemic to all statistical learning algorithms, including SVMs (though, at least in theory, you can get an optimal solution with an SVM). Using such algorithms to learn from past experience (i.e., training data), you generate an approximation function for the underlying distribution you're trying to model. You see the function as worth the time it took to gather training data, select the features, and train, if it approximates pretty well the underlying target distribution you're interested in. You can tell if it pretty well approximates the underlying distribution if it keeps getting things right, and you don't have to keep making excuses for it (say, by saying that "it's complicated").

Anyway, we can view ANNs, SVMs, and other statistical learners as essentially inductive systems, in the sense that, given a set of prior examples, they learn a rule (classifier) that allows us to predict something about new, unseen examples. They generalize, in the sense that unseen examples that match a profile (learned from the training data), even if not an exact fit, may still be classified correctly. It's not a simple one-to-one match. Hence, generalize.

The problem with the generalization performance of all such systems is two-fold. One, they're limited by the set of features that were chosen (a person selects features that are relevant, like "is followed with 'inc.'" for classifying organization mentions in free text). Two, even given optimal feature selection, the generalization performance of inductive systems is always hostage to the information in the training data. We collect examples to use to train the system, and, in the end, we hope that the training data was adequate to model the true distribution (the thing we really want to predict). Problem is, whatever hasn't occurred yet in this true distribution, can't possibly be collected from past examples, and so the entire approach is hostage to the fact that things change in real-life environments, and the larger "pattern" of the true distribution as it unfolds in time may not be captured in the training data. Whenever this happens, the approximation function does not model the true distribution. Bummer.

Now, a feature in particular of such inductive systems (we can substitute "supervised statistical learning sytems" if it sounds more fancy) is this local minima or maxima worry, which I introduced with regard to ANNs, but which is really just a handy way of introducing the general problem of generalizing from past cases to future ones writ large. And it is a problem. Consider time sequence prediction (as opposed to, say, sequence classification such as the well-known document classification task in IR research). In time sequence prediction, the goal is take a sequence of elements at time t, and predict the next element at time t+1. Applying this multiple times you can predict a large sequence of elements through some time n.

And this is where the inductive problem comes in, because if the data you're using to predict the next elements came from some set of prior elements, it's possible that the prior elements (your training data), gave you a model that will get you stuck on a foothill, or, will see the top of a small hill as a bottom valley, and so on. You can't be sure, because the future behavior of the true distribution you don't have. And this is why, in the end, induction--however fancy it gets dressed up in mathematical clothing--not only can be wrong, in theory, but often is, in practice.

We can't see into the future, unfortunately. If we could, we could fix the inductive problem by simply adding in the additional information about the true distribution that we're missing in our training data. But in that case, of course, we'd hardly need the approximation.

Rule Following

"This was our paradox: no course of action could be determined by a rule, because any course of action can be made out to accord with the rule."
Ludwig Wittgenstein, Philosophical Investigations

"If the rule you followed brought you to this, of what use was the rule?"
Anton Chigurh, No Country for Old Men


One the great myths of modern society is that we're following rules to obtain outcomes. I mean rules, roughly, in the sense described here, although I'll also feel free, for these purposes, to equivocate a bit between plans and rules. No harm should be done for now.

So the myth of rule following. We see it in software development (no one seems to notice, or if they do, they dare not mention, that the "rule" was changed a thousand times between conception and completion of project), we see it in the economy, the social sciences, and indeed everywhere that the veneer of science and technology and the almost pathological need for certainty manages to obscure deeper truths about the fragility of our capacities.

Retrodiction, not prediction, is what we're best at, though it is unfortunately and for obvious reasons of little interest. And for various psychological reasons that I'm neither qualified nor interested in researching directly, we're strikingly good at painting failure after failure to predict what comes next with ex post facto explanations that make things just so. Political science is perhaps paradigmatic. It was common in the 1950s to prognosticate about how the (now defunct) USSR would be the preeminent superpower by the 1970s. France (yes, France) was widely thought to be emerging in the 1970s. Japan in the 1980s. China of course today. Our ability to keep proclaiming, generation after generation, our cock-sure predictions about the future state of human societies in the next year, five years, decade (or God forbid, century), is simply amazing, and defies logic. Yet we keep doing it. And we will keep doing it, in spite of all evidence of consistent failure to the contrary.

The psychology of rule following tells us that there's a rule (or a set of rules) that we followed to get to a result (or that will allow us to predict a future result). And when we achieve the result, we tend to confirm the application of the rule, when in fact (chances are) we've made innumerable on-the-fly judgements to get to our result, and then we've tidied things up after the result was achieved by giving credit to the rule. So everything fits. Feels like progress.

On the other side of the coin, when a result is not achieved, instead of recognizing the general problem of using rules, we tend to assume that the particular rule (we claimed to) use, was in fact not adequate. And we set about looking for a new rule, which will of course not be adequate in many contexts too. Such is the nature of our (unexamined) selves. In a deeper and more honest sense we might someday admit that progress (at least in messy, complex situations that we're immersed in), is mostly a function of insights, adaptive thinking as the environment changes, and, well, luck. But we don't see it this way. It doesn't sound like something an expert would say.

So I think that in complex systems (like the weather, or any system where human choice can enter in), our capacity to formulate generalizations that tell us how things will be at time t+n, when we refer to them at time t, is effectively a chimera (whenever n is large enough, which depends on features of the system). Things are constantly new, and different. We formulate plans, and rules, and they guide us, but very loosely, because the environment is constantly in flux. Rules we've grabbed onto "work", only because we keep adjusting things to make them seem to work. The real driver is rather our own wits and insight. And with these much more powerful tools, software does get developed. The Surge in Iraq works. The Space Shuttle (mostly) arrives at the Space Station. And when I get correct directions and follow them, I typically get where I'm going (even if unexpected snags happen). And on and on.

Anyway, in some other post I promise to explain in more depth exactly how rule-following is a mirage, which I haven't yet done (I've asserted mostly only that it is). To be continued. Until then, rest assured that our rules are grains of salt. They just masquerade as so much more.

The Sobriquet

She remained phlegmatic about her sobriquet long after her lover had bestowed it, and then began using it in earnest. But something changed. She noticed first that she was anxious to discuss it when around the table with her girlfriends, and later became horrified at the prospect of its slipping to third parties in public or semi-public moments. She was not a woman prone to obloquy, particularly against her lover, but in almost febrile moments she began to fret that he had drawn her into a miasma of his silly, redolent, refulgent phrases that he thought coruscated their union and she just his style. She didn't know any more. She thought maybe she didn't like her sobriquet. Maybe she hated it.

Exiguous complaints, she countered. And then the frustration would grow until it burst out in fissiparous fragments, and she'd retreat to lying motionless. Unthinking.

In other moments a more psephological mood would emerge, and she'd poll her constant and contradictory thoughts for some majority that might bring solace. Or decision.

Decision. Dissumulation, is all. Dissumulation. He doesn't care, despite his near ubiquitous plaudits. Well nor than does she. It was, she later realized, the codicil that granted her immunity from that death which awaited the other. She was phlegmatic no more.

The Donald

Donald Trump gets interviewed, pretty frequently, by cable news for sound bites on what's happening with the economy, business, banking, this kind of thing. You know, The Donald topics. Well I really do love these Donald moments. This is a guy who once ran for President, promoting himself to the country with memorabilia like "I've had a great time here" (what the heck does that mean? Is this what-happens-in-Vegas campaigning? Sounds like a Chicago politician, with perhaps less criminality and more just standard issue louche).

So he's back on cable news now, giving his wisdom on the auto bailout. The problem with The Donald is that he's so opulently wealthy, so New York Cosmo privileged, that he can't manage the relevant distinctions for us plebian viewers. When asked if he'd buy an American car, he responds that he's got several. In fact, he says, he just had one of his workers buy a Dodge Ram truck for him. He likes Cadillacs, too. Buick makes a good car, he assures us.

The interviewer, Greta Van Suster-whatever, asks if he buys foreign as well, and she should have known better, because (say it with me) of course he does! He buys them all. See, it's just about Donald Trump being really rich. But strangely I still like the guy. He embodies that rarified world of gaudy New York real estate tycoons. He's (weirdly) innocently just that. He tells us with a straight face that he helps the struggling economy by buying expensive things (he's got, strictly speaking, something of a point here.) No time for philanthropic B.S., The Donald's making deals. He just scored another gold statue for one of his homes. Beautiful. Great time to buy em'.

Final thought, The Donald was on I think MSNBC a while back, and was asked about buying real estate. I kid you not, he's on point with a story about how he just picked up a piece of real estate for pennies on the dollar. 112 million is all. It's a great time to buy, he concludes (as if we should all rush out and grab up some resorts in Miami for cheap). The interviewer is slightly exasperated at this point and reminds Mr. Trump that many Americans are struggling to buy a first home, or pay the mortage on an existing. Unfazed. The deals are out there, he says. And they are. If you're The Donald.

The Sea, Part Two

The undulations of the sea lofted him up, and rolled him helplessly down, into the chaotic brine and ocean. He choked. He choked more. His eyes went wide as adrenaline surged, his limbs flailing, then a massive push down into the sea. The spots that grew red in his consciousness grew brighter; his mind reeled and he was strangely cognitive, caught in frantic vascilating between drawing breaths of air or clenching shut to stem so much more choking. He didn't get it right. Water came into his lungs with fantastic pain and no air. No air. He drew in on himself. Where was he? Suddenly, and quit unexpectedly an insight. It should be fine, he thought, suprised at his calmness. Calm. That's what he was forgetting. Calm brought his dreams and his thoughts and then the painless morphinic images where he saw his father. His father was smiling, looking at him. God, his whole life washed over with calm. He was fine. He had always been fine.

The boulder that struck his head got no prize. He had ceased to care, and then he knew no more.

His death lay heavy on the hearts of the living. It was perplexing, and tragic. Perhaps. But perhaps in no one's view in particular, it could be said that his last moments were in fact very much like his first moments, and strangely just so innocent and divine. There was, of course, no one there to say it.

Ode to the 629

I have been traveling since Tuesday and hence no time for posting. But on my return there's my leetle friend, the Smith and Wesson Model 629 .44 Magnum (my wife picked it up from the gun shop while I was gone).

A few comments, in no particular order (actually a false statement; clearly there's a particular order, so hear it here first: all these years folks have been fibbing about the non-particularity of their particular orders).

1. It weighs as much as a kitchen pot half full of water.
2. The chambers are so large that I can get a good portion of my pinky finger into them.
3. Pointing it at yourself in the mirror gives you the heebee geebees (even, obviously, when verifiably unloaded).
4. It's barrel-heavy, with the 6" barrel. Tweaks the wrist a bit. The 6" barrel is I think shortest allowable for hunting. Moose. Or bear. It's Palin Friendly as we now say.
5. The grip is slightly small, although not unreasonably so.
6. It seems for all the world like a gun of such heft that one could make a plausible case that even absent ammo, it could still function as a formidable weapon.
7. My mother-in-law likes it. Really.

The Dry Wit of the Brit

Chistopher Hitchens, you gotta love this guy. He's on Hardball last night with Salon's Joan Walsh debating the Hilary Clinton pick, a perfect platform to launch his everything-but-the kitchen-sink diatribes against the Clintons (he thinks very little of the Clintons, I'm now aware). Ms. Walsh, who apparently was there as a cheerleader for Clinton (needed only the pom poms), and now has a look on her face like Mr. Hitchens is talking about her sister (the camera keeps panning over to her, what high drama), has just about enough and, without engaging the substance of his charges, lobs a Labowski at him: "that's just your opinion Christopher." To which he retorts "...yes, how clever, and look who's saying it? Would you rather I give your opinion?" Walsh just kind of harumphs, but Mathews couldn't resist a chuckle. Neither could I.