The Boot, but not the Heel (Time For Better Search Engines, Part 2)

Well, here we go.

As many people must know, Secretary of the Interior Salazar did say "Our job basically is to keep the boot on the neck of British Petroleum to carry out the responsibilities they have both under the law and contractually to move forward and stop this spill,", which I discovered by throwing words out of the quote (as I should have thought to do); in particular, Rand Paul "improved" it just a bit by adding a heel ("boot heel on the") and changing "neck" to "throat". An ugly image that should have been disavowed by the administration, instead of being repeated, somewhat sheepishly, by Robert Gibbs, the worst presidential press secretary I can remember, at least I can honestly say that has always been my gut feeling.

But it affords an excuse for another lesson on search engines. Voila: http://newstimeline.googlelabs.com/ which I found, described as "experimental" on searching for "search engine" "by date". I had already tinkered with the quote removing the heel, and continuing to use "-rand" "-paul" to find only pages with no mention of rand or paul, and was coming up with Robert Gibbs, then it was Gibbs quoting Salazar.

But the google news timeline really let me do just what I'd been wanting to do, namely find "who said it first". When I did a search for "boot on the neck of bp" using the timeline, I got an array of columns, one per date, with news stories. The bad news is it seems to be limited to news stories from major sources, but it did give a graphic picture of stories containing the phrase blowing up starting on 5/21, when Rand Paul was quoted slightly misquoting the quote. Arrows let me walk back in time -- little or nothing from May 13-20, then a cluster of references going all the way back to May 2, and then stopping.

I find the interface nicely graphic, but slow and cumbersome, and if somebody used the phrase a year or 2 ago, it really wouldn't be much help (correct me, google, if you can). But it did the job, and nicely shows the value of such a feature if we just improve on it a bit, and integrate it into regular google.

It also suggests another class of improvements we could use in our search engines: something approaching search by meaning. Computers can't really "understand" meaning, but they are getting better and better with translation, which indicates quite a bit of adaptation to the structure of language, and so, suppose I could have posed a search like this:

"{boot heel}* on the neck ..." where {...}* means "What's in the bracket or something roughly equivalent".

One problem is, if you have a million exact matches, and a smaller number of modified matches, how to give the user some handle on the variation(s). Typical "search by relevance" arguments would probably see the exact match as way better than the inexact match, so that it would be way, way down on the list. I'd suggest something like, as either an alternative or addition to the current type of google listing, something like:

VARIATIONS:
"boot heel on the throat" 11,707 hits [date range: 5/21-5/23]
"boot heel on the neck..." 1,305 hits [date range: 5/21-5/23] (I'm making the numbers up)
"boot on the neck" 7,222 hits [date range 5/2-5/23]

and then you would click on a variant to see all the specific examples in the format normally used by google, or another search engine.

There is much more to be said about search engines, and vastly much more that I don't know, I'm sure.

The thing about the internet is "The truth is out there", but often, like the dynosaur bone in the rock, it can be quite a job to pry it out of there.

It's Time for Better Search Engines (Who said: "..put my boot heel on the throat of BP")

Is there a search engine that will let me ask?

Who said: "..put my boot heel on the throat of BP"

OK Rand Paul said it, or lets say the the answer looks like a sort of summary "Rand Paul said it" AND a list of pointers to articles quoting Paul as saying it, and maybe a few quite different entries, such as Rand Paul saying he didn't say it. So what if I could say "show me the most atypical entries first". That sounds like a very generally useful followup question when you get 2 million hits, and as far as you can tell the all say more or less the same thing. Could a computer program do a reasonable approximation of what a human (with a year to wade through the 2 million hits) could do? My guess is yes, that wouldn't be a big stretch even.

I've been skimming so many web pages, I feel like I've seen something somewhere quoting someone in Obama's cabinet actually using a phrase like: "..put my boot heel on the throat of BP". Can I confirm that? or be very comfortable in saying it didn't happen (or hear who the Cabinet member was, and see if he/she gets fired the next day)? Well, I can find someone directly attributing the phrase to Obama: "I'll put my boot heel on the throat of BP." Barry Obamma http://twitter.com/busybrains/status/14497679203.

An important question, and perhaps it represents one of those big stories the news media misses: How many people today, next week, next month, next November literally believe or will believe Obama did say that? Are there any pollsters asking that sort of question? My guess, it could easily be something on the order of as many people as think Saddam Hussein was directly behind 9/11 (at least some pollsters paid attention to that).

Relying on existing search engines and their limited abilities, how close could I come to answering this sort of question?
Well if somebody said it before Rand Paul, and Paul picked it up a couple of days later, wouldn't there be some references to this on the web, before there was any association between the phrase and Rand Paul?

Consider this Google search: "put my boot heel on the throat of BP" -paul

The quotes ("") mean I don't want just any combination of the words "put", "my", "boot".... but want that exact phrase. The "-paul" means nothing containing the word "Paul". So I get 4 hits, all from context being clearly from the Ron Paul interview, except for the twitterer directly attributing it to "Barry Obamma".

OK, but what if the quoted secretary was named Paul ____?_____ ?
I tried already -"rand paul", which picked up too many pages in which Rand Paul was just referred to as Paul. Some other approach? OK, when Paul was putting words in the President's mouth, he started with "What I don't like from the president's administration..."

HOW ABOUT: "put my boot heel on the throat of BP" -paul -"What I don't like from"

That cuts the hit count down quite a bit. There are a couple in which "don't" came out "dont" or "donit", or they just cut the quote down so the whole phrase "What I don't like from" didn't appear, and finally we are left with the twitterrer quoting "Barry Obamma" which I'm inclined to discount.

Suppose I could say "Who said it first"? Computer logic to approximate that could rely on that fact that every internet page in google's (or another search engine's) vast database will have a date and time of posting. In fact, can't I just tell google "display in order of posting", which would make the question much more easy to answer? NO, apparently not; at least I don't see how. I could probably put a front end on google accessing google via it's more computer friendly interface (or API), and voila, a new and useful search engine.

For more (and drier) discussions of search engines, see http://en.wikipedia.org/wiki/List_of_search_engines
or searchenginewatch.com,

or just google "search engines". You will get, according to google "About 69,600,000 results". Bon appetite!

Email from a Friend of a Friend: From Urban Legends to Political Smears

Anonymous forwarded emails have for years been a  vehicle for circulating jokes, inspirational pictures and poems, a source of urban legends, and other frivolous but entertaining stuff.


Now, camouflaged by the fluff and amateur political commentary is  a stream of carefully constructed lies and disinformation which does not look like the work of amateurs.

Unlike past, mostly local, whispering campaigns, email is harder to trace and easier to do on a national scale. A couple of years ago, I started getting forwarded emails from my Mom, with claims that could generally be shot down with less than 15 minutes of internet research. They seemed to be really affecting my parents' views, and based on what they told me, they were generally believed by most of their friends. But they were quite simply full of provable lies. They would show signs of having been forwarded a half dozen or so times, with visible 'CC' lists giving them a sort of homey look. When many people receive this sort of thing forwarded by a friend or relative, they are apt to trust it as coming from ordinary outraged citizens as they might not trust direct mass email, but many could simply not have originated as misinformation that the sender believed, which means they can't be anything but deliberately constructed lies, and the number of them, and the similar techniques used seem to prove that they are mass produced.

Here are a couple of references:

The New Right-Wing Smear Machine by Christopher Hayes Oct 25, 2007

MyRightWingDad.net: FW: OBAMA DEATH LIST

If the same sort of phenomenon is going on with Liberal or Ultra Liberal sources, I would be very interested to investigate that as well.

Practical Epistemology

[Heavily revised on 11/3/2011]
Wikipedia defines epistemology as "the branch of philosophy concerned with the nature and scope of knowledge". Traditionally it has led to questions like whether we can really know anything, and discussing the qualities of different kinds of knowledge like logical or mathematical knowledge.

How much attention has been paid, however, to the question "Who can I trust" -- perhaps far and away the most important epistemological question that anyone can ask. Why? Because nearly every bit of knowledge you use to live your life came from some source that you decided to trust.

What does epistemology, the relevant academic discipline, have to say about how we decide who to trust? When I go to Google and pair the word "epistemology" with any of the phrases "who do you trust", "who can you trust", and "who can I trust", the number of "hits" is about 100 for the 1st 2 cases, and 28 for the third. Those who have done more than a few Google searches will recognize how small these numbers are. E.g. if I pair "baseball" with "Who can I trust", I get not not 28 but about 1060 hits. For "music" and "who can I trust", I get 23,300 hits. Is it just that the web has so few references to epistemology? Not really. If I pair "epistemology" with "literary theory" I get 148,000 hits; with "epistemology" and "scientific investigation", I get 25,000 hits; with "mathematics", 1,890,000 hits; with "feminism", 475,000 hits.

So it seems as if I must be either very original, or perhaps very wrongheaded to want to associate the question "Who can I trust" with "epistemology". Yet if I want a true answer to questions like "What needs to be done to make my car run well, and what will it cost?" or "Could this funny looking mole turn into skin cancer?" or "Does this house I'm about to buy have a serious radon problem, or termite infestation?", for the most part I answer these questions by first asking who I can trust to answer them for me -- and in most cases, I'll only ever have at most a superficial idea how the "experts" arrived at their conclusions. So why has epistemology failed to look in that direction?

For a couple of reasons, I think. First, modern philosophy was born out of a reaction to misplaced trust, faith, or dogma. The Church, and especially the Roman Catholic Church, had set itself up as the arbiter of truth. If Galileo said he "saw" moons circling Jupiter, or spots on the Sun, the Bible and/or Aristotle, and their medieval interpretors said no, this was impossible -- and it was dangerous to allow people to claim otherwise.

Then there is the nature of philosophical proof, or demonstration. You have to go through it step by step, and "see" that the first statement implies the second, and so forth, more or less the same way that one can "see" that 2+2=4, or that if A is bigger than B and B is bigger than C, then A is bigger than C. The "demonstration" may have been written on paper by me or by Professor so-and-so, but the implicit premise of such demonstrations is that if it is a valid demonstration, and you are a qualified (at a minimum, "sane") person, it will work when run through your mind. 

Philosophical argument is supposed to be complete in itself. Statements like "Just check X's credentials and you'll see you can trust him" have no place in philosophical reasoning. A philosopher may, like anyone else, think "I should read Dr. S's book because Frank, whom I highly respect, recommends him", or "I'm not going to read this book which claims to be philosophy because the author has no credentials" -- but such decisions are not justified on philosophical grounds, and yet that is just the sort of decision that plays the greatest role in most people's search for the truth, or in particular, for knowledge that directs their actions.

In the process of deciding which doctor I should trust my life to, is there anything that is not subject to doubt? Mr. Smith had a very good outcome with Dr. X, and raves about him, but could Smith just have been lucky? Dr. Y highly recommends Dr. X but mightn't that be due to a close friendship? We may feel that we can "for practical purposes" get around such difficulties, but we cannot prove with philosophical rigor that we made the right choice.

But during the Renaissance and Enlightenment, many people realized there was such an accumulation of misplaced trust, and authorities whose claims were clearly contradicted by the natural world, that it was essential to view arguments on their own internal merit, or based on experiment ("empiricism").


The last few decades have seen a growing consciousness of knowledge as a social phenomenon.  This has, at times, taken on an anti-Enlightenment tone, in the academic world as "postmodernism", and elsewhere as good old anti-intellectualism.  In very recent times, we have seen a new discipline called "Social Epistemology", which addresses some of the questions I've raised here (I've only just learned about it, 18 months after writing the original version of this post).  It seems to be split into two factions, one of which seems too close to postmodernism and Christian intellectualism, but the other, led by Alvin Goldman, looks appealing to me as it purportedly "defends the integrity of truth and shows how to promote it by well-designed forms of social interaction. From science to education, from law to democracy, he shows why and how public institutions should seek knowledge-enhancing practices." **

I do believe we have a need for "knowledge enhancing practices", not to be implemented in some top down fashion, but I believe in the gentler spirit of Amartya Sen and Gene Sharp.  If this last statement makes any sense to you at all, I hope you will write a comment.

** Quote taken from Amazon page for this book: 

The Uses of Pseudo-realism (From The Ontological Comedian, Take 1)

Originally written: Sunday, December 25, 2005

Beware the man who flatters you by saying he's not going to flatter you because you're too smart for that.

Smart persuaders often bank on one's image of oneself as being 'realistic', 'unromantic', etc. If someone is trying to sell you something and says it's a bargain, you'll want to know what's in it for him, and a good salesman will have a good cover story, like the car salesman who says "I just have to make this one more sale by tomorrow to win a trip to Hawaii."

Purveyers of political philosophies, from Randians to Republicans to Marxists will tell you "We're just hard-headed realists, unlike those sentimental utopian Marxists, Democrats, or Saint-Simoneans (or Social Democrats)."

[to be continued]

The Problem of Knowledge

The Real Truth Project is just (for now at least) me thinking about how I know what I know, if I really do know it. And why might my truth and someone else's truth be getting further apart rather than closer together? Why might different groups of people construe the world as if they weren't on the same planet, which as far as I can tell we are? And is there anything to be done about it.

It is an old, old problem, and people have been trying to solve it for at least 2-3 thousand years. I can't say that preliterate people didn't also ask something like this, but their thoughts on the subject had to be more like isolated flashes. Whatever one man or woman thought vanished within a couple of generations or less, unless it became part of the "canon" of the tribe -- what they were able, using special language (like poetry and song) and ritual to preserve in the collective memory of a couple of hundred people with no way to write anything down. So I'll try to stick with surviving writings which go back 2-3 thousand years, and certainly some Greeks of the 5th century BC and thereabouts were asking how do I know what I know, or do I really know it?

It is often said that knowledge is growing at in incredible exponential rate, and sometimes the explosive growth of written "stuff" is trotted out to prove this. I say "written stuff" rather than "written knowledge" because virtually anyone of any point of view will tell you some large percentage of it and maybe all but a tiny fraction, isn't knowledge at all, but is false, or maybe it is just gobbledeegook.

What would a "real" body of knowledge look like? It seems to me it should not be shot through with contradictions, like the body of "religious knowledge" taken collectively, or philosophy for that matter, or "political science".

Descartes in the 17c asked himself "Is there anything that I can't doubt" - he could doubt that he had a body or was standing on solid earth -- he could imagine how these things might be faked. But then he thought, "what is going on -- what has to be going on as I ask myself these questions and try to doubt every thing? Ah!, though is going on. Yes, there it is, going around in my head -- or maybe my head is an illusion, but I can't, by doubting -- a kind of thinking -- eliminate thought as a necessity. It is somewhere, in my head, in the mind of God, or in the giant computer of The Matrix. This "I know I exist because I think" was hailed as a great philosophical achievement. It seemed like some kind of ground to stand on, and ask "what else"?