Every night when Burton went home, he was reminded of how good the everyday Internet had become at connecting dots. “Web 2.0” technologies that encourage people to share information — blogs, photo-posting sites like Flickr or the reader-generated encyclopedia Wikipedia — often made it easier to collaborate with others. When the Orange Revolution erupted in Ukraine in late 2004, Burton went to Technorati, a search engine that scours the “blogosphere,” to find the most authoritative blog postings on the subject. Within minutes, he had found sites with insightful commentary from American expatriates who were talking to locals in Kiev and on-the-fly debates among political analysts over what it meant. Because he and his fellow spies were stuck with outdated technology, they had no comparable way to cooperate — to find colleagues with common interests and brainstorm online.
By the late summer, Fingar decided the Intellipedia experiment was sufficiently successful that he would embark on an even more high-profile project: using Intellipedia to produce a “national intelligence estimate” for Nigeria. An N.I.E. is an authoritative snapshot of what the intelligence community thinks about a particular state — and a guide for foreign and military policy. Nigeria, Fingar said, is a complex country, with issues ranging from energy to Islamic radicalism to polio outbreaks to a coming election. Intellipedia’s Nigeria page will harness the smarts of the dozen or so analysts who specialize in the country. But it will also, Fingar hopes, attract contributions from other intelligence employees who have expertise Fingar isn’t yet aware of — an analyst who served in the Peace Corps in Nigeria, or a staff member who has recently traveled there. In the traditional method of producing an intelligence estimate, Fingar said, he would call every agency and ask to borrow their Africa expert for a week or two of meetings. “And they’d say: ‘Well, I only got one guy who can spell Nigeria, and he’s traveling. So you lose.’ ” In contrast, a wiki will “change the rules of who can play,” Fingar said, since far-flung analysts and agents around the world could contribute, day or night.
Intellipedia hopes to attract contributions from intelligence employees who have expertise the agency isn't yet aware of — an analyst who served in the Peace Corps in Nigeria, or a staff member who has recently traveled there
Open-Source Spying
By CLIVE THOMPSON
Published: December 3, 2006
[Excerpt]
When Matthew Burton arrived at the Defense Intelligence Agency in January 2003, he was excited about getting to his computer. Burton, who was then 22, had long been interested in international relations: he had studied Russian politics and interned at the U.S. consulate in Ukraine, helping to speed refugee applications of politically persecuted Ukrainians. But he was also a big high-tech geek fluent in Web-page engineering, and he spent hours every day chatting online with friends and updating his own blog. When he was hired by the D.I.A., he told me recently, his mind boggled at the futuristic, secret spy technology he would get to play with: search engines that can read minds, he figured. Desktop video conferencing with colleagues around the world. If the everyday Internet was so awesome, just imagine how much better the spy tools would be.
But when he got to his cubicle, his high-tech dreams collapsed. “The reality,” he later wrote ruefully, “was a colossal letdown.”
The spy agencies were saddled with technology that might have seemed cutting edge in 1995. When he went onto Intelink — the spy agencies’ secure internal computer network — the search engines were a pale shadow of Google, flooding him with thousands of useless results. If Burton wanted to find an expert to answer a question, the personnel directories were of no help. Worse, instant messaging with colleagues, his favorite way to hack out a problem, was impossible: every three-letter agency — from the Central Intelligence Agency to the National Security Agency to army commands — used different discussion groups and chat applications that couldn’t connect to one another. In a community of secret agents supposedly devoted to quickly amassing information, nobody had even a simple blog — that ubiquitous tool for broadly distributing your thoughts.
Something had gone horribly awry, Burton realized. Theoretically, the intelligence world ought to revolve around information sharing. If F.B.I. agents discover that Al Qaeda fund-raising is going on in Brooklyn, C.I.A. agents in Europe ought to be able to know that instantly. The Internet flourished under the credo that information wants to be free; the agencies, however, had created their online networks specifically to keep secrets safe, locked away so only a few could see them. This control over the flow of information, as the 9/11 Commission noted in its final report, was a crucial reason American intelligence agencies failed to prevent those attacks. All the clues were there — Al Qaeda associates studying aviation in Arizona, the flight student Zacarias Moussaoui arrested in Minnesota, surveillance of a Qaeda plotting session in Malaysia — but none of the agents knew about the existence of the other evidence. The report concluded that the agencies failed to “connect the dots.”
By way of contrast, every night when Burton went home, he was reminded of how good the everyday Internet had become at connecting dots. “Web 2.0” technologies that encourage people to share information — blogs, photo-posting sites like Flickr or the reader-generated encyclopedia Wikipedia — often made it easier to collaborate with others. When the Orange Revolution erupted in Ukraine in late 2004, Burton went to Technorati, a search engine that scours the “blogosphere,” to find the most authoritative blog postings on the subject. Within minutes, he had found sites with insightful commentary from American expatriates who were talking to locals in Kiev and on-the-fly debates among political analysts over what it meant. Because he and his fellow spies were stuck with outdated technology, they had no comparable way to cooperate — to find colleagues with common interests and brainstorm online.
Burton, who has since left the D.I.A., is not alone in his concern. Indeed, throughout the intelligence community, spies are beginning to wonder why their technology has fallen so far behind — and talk among themselves about how to catch up. Some of the country’s most senior intelligence thinkers have joined the discussion, and surprisingly, many of them believe the answer may lie in the interactive tools the world’s teenagers are using to pass around YouTube videos and bicker online about their favorite bands. Billions of dollars’ worth of ultrasecret data networks couldn’t help spies piece together the clues to the worst terrorist plot ever. So perhaps, they argue, it’ s time to try something radically different. Could blogs and wikis prevent the next 9/11?
The job of an analyst used to be much more stable — even sedate. In the ’70s and ’80s, during the cold war, an intelligence analyst would show up for work at the C.I.A.’s headquarters in Langley, Va., or at the National Security Agency compound in Fort Meade, Md., and face a mess of paper. All day long, tips, memos and reports from field agents would arrive: cables from a covert-ops spy in Moscow describing a secret Soviet meeting, or perhaps fresh pictures of a missile silo. An analyst’s job was to take these raw pieces of intelligence and find patterns in the noise. In a crisis, his superiors might need a quick explanation of current events to pass on to their agency heads or to Congress. But mostly he was expected to perform long-term “strategic analysis” — to detect entirely new threats that were still forming.
And during the cold war, threats formed slowly. The Soviet Union was a ponderous bureaucracy that moved at the glacial speed of the five-year plan. Analysts studied the emergence of new tanks and missiles, pieces of hardware that took years to develop. One year, an analyst might report that the keel for a Soviet nuclear submarine had been laid; a few years later, a follow-up report would describe the submarine’s completion; even more years later, a final report would detail the sea trials. Writing reports was thus a leisurely affair, taking weeks or months; thousands of copies were printed up and distributed via interoffice mail. If an analyst’s report impressed his superiors, they’d pass it on to their superiors, and they to theirs — until, if the analyst was very lucky, it landed eventually in the president’s inner circle. But this sort of career achievement was rare. Of the thousands of analyst reports produced each year, the majority sat quietly gathering dust on agency shelves, unread by anyone.
[Excerpt]
There was already one digital pipeline that joined the agencies (though it had its own limitations): Intelink, which connects most offices in each intelligence agency. It was created in 1994 after C.I.A. officials saw how the Web was rapidly transforming the way private-sector companies shared information. Intelink allows any agency to publish a Web page, or put a document or a database online, secure in the knowledge that while other agents and analysts can access it, the outside world cannot.
So why hasn’t Intelink given young analysts instant access to all secrets from every agency? Because each agency’s databases, and the messages flowing through their internal pipelines, are not automatically put onto Intelink. Agency supervisors must actively decide what data they will publish on the network — and their levels of openness vary. Some departments have created slick, professional sites packed full of daily alerts and searchable collections of their reports going back years. Others have put up little more than a “splash page” announcing they exist. Operational information — like details of a current covert action — is rarely posted, usually because supervisors fear that a leak could jeopardize a delicate mission.
Nonetheless, Intelink has grown to the point that it contains thousands of agency sites and several hundred databases. Analysts at the various agencies generate 50,000 official reports a year, many of which are posted to the network. The volume of material online is such that analysts now face a new problem: data overload. Even if they suspect good information might exist on Intelink, it is often impossible to find it. The system is poorly indexed, and its internal search tools perform like the pre-Google search engines of the ’90s.“
One of my daily searches is for words like ‘Afghanistan’ or ‘Taliban,’ ” I was told by one young military analyst who specializes in threats from weapons of mass destruction. (He requested anonymity because he isn’t authorized to speak to reporters.) “So I’m looking for reports from field agents saying stuff like, ‘I’m out here, and here’s what I saw,’ ” he continued. “But I get to my desk and I’ve got, like, thousands a day — mountains of information, and no way to organize it.”
Adding to the information glut, there’s an increasingly large amount of data to read outside of Intelink. Intelligence analysts are finding it more important to keep up with “open source” information — nonclassified material published in full public view, like newspapers, jihadist blogs and discussion boards in foreign countries. This adds ever more calories to the daily info diet. The W.M.D. analyst I spoke to regularly reads the blog of Juan Cole, a University of Michigan professor known for omnivorous linking to, and acerbic analysis of, news from the Middle East. “He’s not someone spies would normally pay attention to, but now he’s out there — and he’s a subject-matter expert, right?” the analyst said.
Intelligence hoarding presented one set of problems, but pouring it into a common ocean, Meyerrose realized soon after moving into his office, is not the answer either. “Intelligence is about looking for needles in haystacks, and we can’t just keep putting more hay on the stack,” he said. What the agencies needed was a way to take the thousands of disparate, unorganized pieces of intel they generate every day and somehow divine which are the most important.
Intelligence heads wanted to try to find some new answers to this problem. So the C.I.A. set up a competition, later taken over by the D.N.I., called the Galileo Awards: any employee at any intelligence agency could submit an essay describing a new idea to improve information sharing, and the best ones would win a prize. The first essay selected was by Calvin Andrus, chief technology officer of the Center for Mission Innovation at the C.I.A. In his essay, “The Wiki and the Blog: Toward a Complex Adaptive Intelligence Community,” Andrus posed a deceptively simple question: How did the Internet become so useful in helping people find information?
Andrus argued that the real power of the Internet comes from the boom in self-publishing: everyday people surging online to impart their thoughts and views. He was particularly intrigued by Wikipedia, the “reader-authored” encyclopedia, where anyone can edit an entry or create a new one without seeking permission from Wikipedia’s owners. This open-door policy, as Andrus noted, allows Wikipedia to cover new subjects quickly. The day of the London terrorist bombings, Andrus visited Wikipedia and noticed that barely minutes after the attacks, someone had posted a page describing them. Over the next hour, other contributors — some physically in London, with access to on-the-spot details — began adding more information and correcting inaccurate news reports. “You could just sit there and hit refresh, refresh, refresh, and get a sort of ticker-tape experience,” Andrus told me. What most impressed Andrus was Wikipedia’s self-governing nature. No central editor decreed what subjects would be covered. Individuals simply wrote pages on subjects that interested them — and then like-minded readers would add new facts or fix errors. Blogs, Andrus noted, had the same effect: they leveraged the wisdom of the crowd. When a blogger finds an interesting tidbit of news, he posts a link to it, along with a bit of commentary. Then other bloggers find that link and, if they agree it’s an interesting news item, post their own links pointing to it. This produces a cascade effect. Whatever the first blogger pointed toward can quickly amass so many links pointing in its direction that it rockets to worldwide notoriety in a matter of hours.
Spies, Andrus theorized, could take advantage of these rapid, self-organizing effects. If analysts and agents were encouraged to post personal blogs and wikis on Intelink — linking to their favorite analyst reports or the news bulletins they considered important — then mob intelligence would take over. In the traditional cold-war spy bureaucracy, an analyst’s report lived or died by the whims of the hierarchy. If he was in the right place on the totem pole, his report on Soviet missiles could be pushed up higher; if a supervisor chose to ignore it, the report essentially vanished. Blogs and wikis, in contrast, work democratically. Pieces of intel would receive attention merely because other analysts found them interesting. This grass-roots process, Andrus argued, suited the modern intelligence challenge of sifting through thousands of disparate clues: if a fact or observation struck a chord with enough analysts, it would snowball into popularity, no matter what their supervisors thought.
A profusion of spy blogs and wikis would have another, perhaps even more beneficial impact. It would drastically improve the search engines of Intelink. In a paper that won an honorable mention in the Galileo Awards, Matthew Burton — the young former D.I.A. analyst — made this case. He pointed out that the best Internet search engines, including Google, all use “link analysis” to measure the authority of documents. When you type the search “Afghanistan” into Google, it finds every page that includes that word. Then it ranks the pages in part by how many links point to the page — based on the idea that if many bloggers and sites have linked to a page, it must be more useful than others. To do its job well, Google relies on the links that millions of individuals post online every day.
This, Burton pointed out, is precisely the problem with Intelink. It has no links, no social information to help sort out which intel is significant and which isn’t. When an analyst’s report is posted online, it does not include links to other reports, even ones it cites. There’s no easy way for agents to link to a report or post a comment about it. Searching Intelink thus resembles searching the Internet before blogs and Google came along — a lot of disconnected information, hard to sort through. If spies were encouraged to blog on Intelink, Burton reasoned, their profuse linking could mend that situation. “
[Excerpt]
Fingar and Wertheimer are now testing whether a wiki could indeed help analysts do their job. In the fall of 2005, they joined forces with C.I.A. wiki experts to build a prototype of something called Intellipedia, a wiki that any intelligence employee with classified clearance could read and contribute to. To kick-start the content, C.I.A. analysts seeded it with hundreds of articles from nonclassified documents like the C.I.A. World Fact Book. In April, they sent out e-mail to other analysts inviting them to contribute, and sat back to see what happened.
By this fall, more than 3,600 members of the intelligence services had contributed a total of 28,000 pages. Chris Rasmussen, a 31-year-old “knowledge management” engineer at the National Geospatial-Intelligence Agency, spends part of every day writing or editing pages. Rasmussen is part of the younger generation in the intelligence establishment that is completely comfortable online; he regularly logs into a sprawling, 50-person chat room with other Intellipedians, and he also blogs about his daily work for all other spies to read. He told me the usefulness of Intellipedia proved itself just a couple of months ago, when a small two-seater plane crashed into a Manhattan building. An analyst created a page within 20 minutes, and over the next two hours it was edited 80 times by employees of nine different spy agencies, as news trickled out. Together, they rapidly concluded the crash was not a terrorist act. “In the intelligence community, there are so many ‘Stay off the grass’ signs,” Rasmussen said. “But here, you’re free to do what you want, and it works.”
[Excerpt]
And the critics might turn out to be right. As Clay Shirky of N.Y.U. points out, most wikis and blogs flop. A wiki might never reach a critical mass of contributors and remain anemic until eventually everyone drifts away; many bloggers never attract any attention and, discouraged, eventually stop posting. Wikipedia passed the critical-mass plateau a year ago, but it is a rarity. “The normal case for social software is failure,” Shirky said. And because Intellipedia is now a high-profile experiment with many skeptics, its failure could permanently doom these sorts of collaborative spy endeavors.
