There are many who are skeptical not least because of the digital divide: “You mean the pulse of the Data Have’s? The pulse of the affluent?” These rhetorical questions are perfectly justified, which is why social media alone should not be the sole source of information that feeds into decision-making for policy purposes. But millions are joining the social media ecosystem everyday, so the selection bias is not increasing but decreasing. We may not be able to capture the pulse of the planet comprehensively at a very high resolution yet, but the pulse of the majority of the world is certainly growing louder by the day.

Planet Earth by night. Credit: NASA.

This map of the world at night (based on 2011 data) reveals areas powered by electricity. Yes, Africa has far less electricity consumption. This is not misleading, it has been shown to be an accurate proxy for industrial development (amongst other indexes). Does this data suffer from selection bias? Yes, the data is biased towards larger cities rather than the long tail. Does this render the data and map useless? Hardly. It all depends on what the question is.

Live map of Tweets. Credit: Franck Ernewein.

What if our world were lit up by information instead of lightbulbs? The map above from TweetPing does just that. The website displays tweets in real-time as they’re posted across the world. Strictly speaking, the platform displays 10% of the ~340 million tweets posted each day (i.e., the “Decahose” rather than the “Firehose”). But the volume and velocity of the pulsing ten percent is already striking.

Geo-tagged Tweets and Flickr pictures. Credit: Eric Fischer.

One may think this picture depicts electricity use in Europe. Instead, this is a map of geo-located tweets (blue dots) and Flickr pictures (red dots). “White dots are locations that have been posted to both” (2). The number of active Twitter users grew 40% in 2012, making Twitter the fastest growing social network on the planet. Over 20% of the world’s internet population is now on Twitter (3). The Sightsmap below is a heat map based on the number of photographs submitted to Panoramio at different locations.

The map below depicts friendship ties on Facebook. This was generated using data when there were “only” 500 million users compared to today’s 1 billion+.

Friendship links on Facebook. Credit: Facebook.

The map below does not depict electricity use in the US or the distribution of the population based on the most recent census data. Instead, this is a map of check-in’s on Foursquare. What makes this map so powerful is not only that it was generated using 500 million check-in’s but that “all those check-ins you see aren’t just single points—they’re links between all the other places people have been.”

Location of Foursquare check-in’s. Credit: Foursquare.

TwitterBeat takes the (emotional) pulse of the planet by visualizing the Twitter Decahose in real-time using sentiment analysis. The crisis map in the YouTube video below comprises all tweets about Hurricane Sandy over time. “[Y]ou can see how the whole country lights up and how tweets don’t just move linearly up the coast as the storm progresses, capturing the advance impact of such a large storm and its peripheral effects across the country” (4).

These social media maps don’t only “work” at the country level or for Western industrialized states. Take the following map of Jakarta made almost exclusively from geo-tagged tweets. You can see the individual roads and arteries (nervous system). Granted, this map works so well because of the horrendous traffic but nevertheless a pattern emerges, one that is strongly correlated to Jakarta’s road network. And unlike the map of the world at night, we can capture this pulse in real time and at a fraction of the cost.

Geo-tagged Tweets in Jakarta. Credit: Eric Fischer.

Like any young nervous system, our social media system is still growing and evolving. But it is already adding value. The analysis of tweets predicts the flu better than the crunching of traditional data used by public health institutions, for example. And the analysis of tweets from Indonesia also revealed that Twitter data can be used to monitor food security in real-time.

The main problem I see about all this has much less to do with issues of selection bias and unrepresentative samples, etc. More problematic is the centralized nature of this overall data and the fact that it is closed data. Yes, the above maps are public, but the underlying data is not. In their new study, “The Politics of Twitter Data,” Cornelius Puschmann and Jean Burgess argue that the “owners” of social media data are the platform providers, not the end users. Yes, access to Twitter.com and Twitter’s API is free but end users are limited to downloading just a few thousand tweets per day. (For comparative purposes, more than 20 million tweets were posted during Hurricane Sandy). Getting access to more data can cost hundreds of thousands of dollars. In other words, as Puschmann and Burgess note, “only corporate actors and regulators—who possess both the intellectual and financial resources to succeed in this race—can afford to participate,” which means “that the emerging data market will be shaped according to their interests.”

“Social Media: Pulse of the Planet?” It’s getting there, but only a few elite doctors can take the full pulse in real-time.

Patrick Meier is a 2012 National Geographic Emerging Explorer. He is an internationally recognized thought leader on the application of new technologies for positive social change. He currently serves as Director of Social Innovation at the Qatar Foundation’s Computing Research Institute (QCRI). Patrick also authors the widely respected iRevolution blog and tweets at @patrickmeier.

]]> http://newswatch.nationalgeographic.com/2013/02/05/social-media-pulse-of-the-planet/feed/ 5 http://newswatch.nationalgeographic.com/2012/12/19/digital-disaster-response/ http://newswatch.nationalgeographic.com/2012/12/19/digital-disaster-response/#comments Wed, 19 Dec 2012 15:55:54 +0000 Patrick Meier http://newswatch.nationalgeographic.com/?p=73797 The United Nations Office for the Coordination of Humanitarian Affairs (OCHA) activated the Digital Humanitarian Network (DHN) on December 5th at 3pm Geneva time (9am New York). The activation request? To collect all relevant tweets about Typhoon Pablo posted on December 4th and 5th; identify pictures and videos of damage/flooding shared in those tweets; geo-locate, time-stamp and categorize this content. The UN requested that this database be shared with them by 5am Geneva time the following day.

Damage caused by Typhoon Pablo. Credit: CBCPNews.com.

The DHN is composed of several members who form Solution Teams when the network is activated. The purpose of Digital Humanitarians is to support humanitarian organizations in their disaster response efforts around the world. Solution Team volunteers analyzed over 20,000 tweets in just 10  hours using a variety of methods ranging from automated algorithms to micro-tasking. They used Geofeedia to identify all relevant pictures/videos that were already geo-tagged by users. About a dozen were identified in this manner. They also partnered with the Qatar Foundation Computing Research Institute’s (QCRI) Crisis Computing Team to collect all tweets posted on December 5th with the hashtags endorsed by the Philippine Government. QCRI ran algorithms on the dataset to remove (1) all retweets and (2) all tweets without links (URLs).

Geofeedia used in response to Typhoon Pablo. Credit: Geofeedia.com.

On the micro-tasking side, digital volunteers used PyBossa, a free and open-source micro-tasking platform. Micro-tasking envolves taking a big task and turning it into a series of smaller, more manageable tasks. Colleagues at PyBossa rapidly customized a platform to support the Digital Humanitarian Network. Volunteers would simply go to the PyBossa website for Typhoon Pablo where they’d be shown one tweet at time as the screenshot below demonstrates.

The micro-tasking platform PyBossa in use for the response to Typhoon Pablo. Credit: PyBossa.

The result? Within 10 hours, over 20,000 tweets were analyzed using a combination of methodologies. By 4.30am Geneva time, the efforts of the Digital Humanitarian Network resulted in a database of 138 highly annotated tweets. Just hours later, UN OCHA published the map below which is entirely sourced from the social media analysis produced by the Digital Humanitarian Network. This is the first ever map of this kind and an important milestone in digital humanitarian response. (Simply click on the map to enlarge).

Official UN social media crisis map of Typhoon Pablo. Credit: OCHA.

Just a few days ago, the Digital Humanitarian Network was once again activated; this time in response to Cyclone Evan in Samoa. If you want to become a digital humanitarian, please visit the DHN website to learn more about the individual teams and contact those you are most interested in.

Patrick Meier is a 2012 National Geographic Emerging Explorer. He is an internationally recognized thought leader on the application of new technologies for positive social change. He currently serves as Director of Social Innovation at the Qatar Foundation’s Computing Research Institute (QCRI). Patrick also authors the widely respected iRevolution blog and tweets at @patrickmeier.

National Geographic digital exploration of Mongolia. Credit: National Geographic.

More than 6 million square kilometers of satellite imagery is produced every single day. The total surface area of the moon is about 35 million square kilometers. So every 6 days, there is a new moon’s worth of exploration to be done in the digital world of satellite imagery. Question is, how can we possibly explore an entire new moon every week? Do we need to build the digital equivalent of the Millenium Falcon? Do we even have a pilot good enough for this mission? The software to automatically and accurately analyze this vast amount of satellite imagery is still not good enough. So what to do? Turns out National Geographic had the answer all along: crowdsource the expedition.

In their phenomenal project, Valley of the Kahns, National Geographic crowdsourced the  analysis of high resolution satellite imagery in the search for clues to an 800 year old mystery, the location of Genghis Khan’s tomb. Welcome to my world, the world of intrepid digital exploration. The answer to the 3 million kilometer question wasn’t one pilot and one Millennium Falcon. No! The answer was hundreds of thousands of pilots from all around the world flying their own X-wings over the vast virtual landscape of Mongolia in search of Genghis Khan.

I blogged about this awesome project when it was launched and described how we could take this same approach in humanitarian crises. A few months later, the crisis in the Horn of Africa began to escalate, displacing a massive number of peoples West of Mogadishu. So the UN Refugee Agency (UNHCR) asked me if we could use the same approach as National Geographic’s to estimate the displaced refugee population in Somalia. Why? Because due to Al Shabbab’s terrorist activities, humanitarians could not do this kind of survey on the ground, they too were being targeted and kidnapped. So we needed to take it to the skies and the UN was in dire need of pilots.

Tomnod was customized to support the United Nations in Somalia. Credit: Tomnod.

So we used the same technology that was used for the Mongolia expedition. Called Tomnod, the platform is designed to crowdsource and crowdtag satellite imagery. We obtained free imagery from DigitalGlobe, which was then “sliced up” into thousands of smaller pictures. Each of these, like the one below was then analyzed by digital explorers looking for signs of permanent and temporary shelters.

The satellite imagery was sliced up into small pictures. Credit: Tomnod.com.

Whey found such shelters, they would simply tag the feature with the appropriate icon, just like in the Valley of the Khans. Only when a shelter was tagged by at least three individual volunteers would that data point be shared with the United Nations. This was a great way to ensure some quality control in the process.

The result? Within 120 hours, volunteers created over a quarter million tags after analyzing close to 4,000 individual images, thus yielding a triangulated count of some 47,000 shelters in the Afgooye corridor of Somalia, which the UN could use to estimate the population in the area. To provide context, it took two UNHCR staff over an entire month to do this back in 2010. We did the equivalent in 120 hours.

Tomnod uses a consensus based algorithm to triangulate the tagging. Credit: Tomnod.

But who do I mean when I say “we did this”? I mean the Standby Volunteer Task Force (SBTF), a global network of some 800+ volunteers in 80+ countries who support humanitarian and human rights organizations in times of need. They were the pilots who flew the Somalia Mission for the United Nations after being inspired by National Geographic’s search for Genghis Khan in Mongolia.

Digital expeditions like these can democratize the next frontier of exploration. So I look forward to collaborating with my colleagues at National Geographic to support future explorations into the digital unknown. Onwards!

Patrick Meier is a 2012 National Geographic Emerging Explorer. He is an internationally recognized thought leader on the application of new technologies for positive social change. He currently serves as Director of Social Innovation at the Qatar Foundation’s Computing Research Institute (QCRI). Patrick also authors the widely respected iRevolution blog and tweets at @patrickmeier.

The National Geographic Society has a long history of crisis mapping disasters. But what happened in Haiti on January 12, 2010 would forever change the very concept of a crisis map. A devastating earthquake struck the country’s capital that Tuesday afternoon. I was overwhelmed with emotions when I heard the news just an hour later. Over 100,000 people were feared dead. Some very close friends of mine were doing research in Port-au-Prince at the time and I had no idea whether they had survived the earthquake. So I launched a live crisis map of Haiti. But this was an emotional reaction rather than a calculated plan with a detailed strategy. I was in shock and felt the need to do something, anything. It was only after midnight that I finally got an SMS reply from my friends. They had narrowly escaped a collapsing building. But many, many others were not near as lucky. I continued mapping.

Sourcing the Map

This is what the map looked liked after midnight on January 13th. What was I mapping exactly? Tweets. I had found a dozen Haitians tweeting live from Port-au-Prince shortly after the earthquake. They were describing scenes of devastation but also hope as this tweet shows:

A tweet reporting that a drug store had reopened. Credit: Screenshot from Twitter.

I was using the Ushahidi platform, a free and open source mapping technology from Africa. Think of Ushahidi, which means “witness” in Swahili, as a multi-media inbox connected to a live map. I added these Twitter users to my inbox and began mapping the most urgent Tweets (those that had enough geographic information to be mapped). The following night, several friends joined me in the living room of my dorm to help map Haiti’s needs.

Volunteers to the Rescue

But within a couple days, we couldn’t keep up with the vast amount of information being reported via both social media and mainstream media. So I reached out to friends at The Fletcher School (Tufts University) where I was doing my PhD. By the end of the week, we had trained over 100 graduate and undergraduate students on how to monitor social and mainstream media for relevant, mappable content. These “digital humanitarians” began to manually monitor hundreds and hundreds of online sources for information on Haiti almost 24/7. The Ushahidi Haiti Crisis Map became a live map with some 2,000 individual reports added during the entire project.

Close up of the Haiti Map. Each number represents the individual number of reports within the area. Users could zoom in further to see the individual reports. Credit: Ushahidi Haiti Project (UHP).

Using Satellite Imagery

But mapping this content became more and more challenging because Port-au-Prince was half missing on the the Google Map of Haiti. The city and roads had not been fully mapped by Google Inc. So some colleagues at OpenStreetMap crowdsourced the most detailed roadmap of Haiti ever produced in just a matter of days. They used satellite imagery provided by the World Bank to carefully trace the road network onto an OpenStreetMap of Haiti. In fact, hundreds of volunteers from all around the world collaborated in these efforts. The following video is an animation of this tracing in action. Over 1.4 million edits/traces (flashes of light in the video) were made to the map in just a matter of weeks.

Crowdsourcing via SMS

Just hours after launching the crisis map, we also set up an international SMS number that members of the Haitian Diaspora could text important reports for us to map. The next day, my colleague Josh Nesbit from Medic Mobile started looking for local SMS options to support our Ushahidi Haiti Project:

Incredibly, someone following his Twitter feed in Cameroon put him in touch with a colleague who was working at Digicel, the largest telecommunications company in Haiti. Within days, we had secured a toll-free SMS number (4636) that allowed anyone in Haiti to text in their most urgent needs and location. This was made possible thanks to multiple groups: Thomson-Reuters Foundation, InSTEDD, US State Department and of course Digicel. In the days that followed, colleagues in Port-au-Prince got the word out about this SMS number by visiting several local community radio stations. They also explained that this was only an information service, not a humanitarian hotline.

Volunteers live mapping the crisis at the Ushahidi Haiti Situation Room in the basement of The Fletcher School. Credit: Ushahidi Haiti Project (UHP).

Translating Needs

Soon enough, we began receiving thousands of text messages. As expected, these SMS’s were written in Haitian Creole rather than English. So my Ushahidi colleague Brian Herbert pulled an all-nighter to customize a web-based interface which was hosted at http://4636.Ushahidi.com [no longer online]. The purpose of this second platform was to enable Haitian-Creole speaking volunteers to translate and geo-locate text messages sent to the SMS number. In effect, the platform was designed to enable the crowdsourced translation and geo-location of incoming text messages specifically for the Ushahidi Haiti Project. Many volunteers from the Haitian Diaspora joined the cause after hearing about the need for volunteers via Facebook.

Some Concerns

During this time, we contacted severals lawyers in Boston to determine whether we could even map these text messages from a privacy standpoint. They opined that we had implicit consent. Two seasoned humanitarian colleagues were also consulted for their feedback. They noted that Haiti was a particularly low risk situation. In other words, the possible harm that could come to local populations was minimal. Do No Harm is a standard principle for anyone operating in a humanitarian crisis or recovery context. The principle is important because it recognizes there is a risk to any intervention. And what is critical in deciding whether a certain course of action is ethical is to consider whether it could potentially cause harm to the local population. Given this feedback, we collectively decided that making the data open in this case was of minimal risk.

Meanwhile, back on the World Wide Web, several hundred volunteers logged on to Ushahidi’s translation platform and reportedly translated some 80,000 10,000 text messages during the first few weeks. Most of these, however, were either not relevant, actionable or mappable. So volunteers at The Fletcher School triaged the translated messages and only mapped the most important life and death messages, i.e., less than 2% of all SMS’s, a very small percentage indeed.

Engaging the Diaspora

Since a number of these SMS’s required more precise geo-location before they could be added to the map, we worked closely with many members of the Haitian Diaspora in Boston who obviously knew their country far better than any of us did. A number of our Haitian friends actually joined us (often for long hours on end) in our make-shift “Situation Room” at The Fletcher School (picture below).

Haitian volunteers crisis mapping Haiti with Sabina Carlson in the Haiti Situation Room at The Fletcher School. Sabina, who speak fluent Creole, was the project's volunteer liaison for the Haitian Diaspora. Credit: Ushahidi.

Operational Response

On January 19th, just a week after the earthquake, someone from the US Coast Guard emailed us with the following question: “I am compiling reports from Haiti for the US Coast Guard and Joint Task Force Command Center. Is there someone I can speak with about how better to use the information in Ushahidi?” Several days later, we set up a dedicate Skype chat with the Coast Guard to fast-forward the most urgent (and actionable) content that was being added to the live Haiti Crisis Map. We were also contacted by an American Search and Rescue team in Port-au-Prince who urgently needed GPS coordinators for the locations of trapped individuals. More on this incredible story here.

On January 22nd, the US Marine Corps got in touch with us via email:

“I am with the US Marine Corps. I am stateside assisting the 22 MEU [Marine Expeditionary Unit] coming off the USS Bataan [on the Haitian Coast]. We want to use your data to bring aid to the people Haiti right now. The USMC is focusing on Leogane, Grand Goave, and Petit Goave. Is there a way to import your data into Google Earth or GIS? We want to make this work for the people of Haiti…please let me know ASAP”

Needless to say, we replied ASAP. On the same day, FEMA’s Administrator Craig Fugate,  shared the following on his Twitter feed:

Craig Fugate is the Administrator. Credit: Screenshot from Twitter.

Five days later, the same contact from the US Marine Corps shared the following by email (which we got permission to make public):

“I can not overemphasize to you what the work of the Ushahidi/Haiti has provided. It is saving lives every day. I wish I had time to document to you every example, but there are too many and our operation is moving too fast.  Here is one from the 22 MEU [Marine Expeditionary Unit]: ‘We had data on an area outside of Grand Goave needing help. Today, we sent an assessment team out there to validate their needs and everything checked out. While the team was out there, they found two old women and a young girl with serious injuries from the earthquake; one of the women had critical respiratory issues. They were evacuated.’

Your site saved these people’s lives. I say with confidence that there are 100s of these kind of stories. The Marine Corps is using your project every second of the day to get aid and assistance to the people that need it most. We did have a tech barrier that we had to surmount. The Marines downrange have Google Earth and your site does not work on the ship for them. So, I had Georgia Tech create a bridge from your site to Google Earth.

But it is YOUR data and YOUR work that is putting aid and assistance directly on the target and saving lives. Our big gap right now is locating NGOs and where they are working. Your site is helping with that. Keep up the good work!! You are making the biggest difference of anything I have seen out there in the open source world.”

Onwards

These incredible efforts following the Haiti earthquake demonstrated a huge potential for the future of humanitarian response. Student volunteers in Boston working online with the Diaspora using free mapping technology from Africa could help save lives in another country thousands of miles away without ever setting foot in said country. In time, these reactive and organic volunteer-driven efforts in Haiti, and those that followed that same year in Chile, Pakistan and Russia, led to the launch of the award-winning Standby Volunteer Task Force (SBTF), a global network of 850+ volunteers in more than 80 countries around the world who use their live mapping skills to support humanitarian, human rights, development and media organizations .

The purpose of the SBTF is to create a network of already trained volunteers so we don’t have to scramble again like in Haiti. SBTF volunteers (or Mapsters as they are called) have since partnered with multiple organizations in dozens of deployments around the world.  The SBTF is now also part of the Digital Humanitarian Network (DHNetwork), a new consortium of volunteer networks who support humanitarian organizations.

For my next blog post in this series on Crisis Mapping, I’ll share an equally remarkable story about the United Nations, Genghis Khan, Satellite Imagery, Somalia and National Geographic. Yes, they’re all  connected in an intriguing way. Stay tuned to find out how! And remember, to map the world is to know it. But to map the world live is to change it before it’s too late.

Patrick Meier is a 2012 National Geographic Emerging Explorer. He is an internationally recognized thought leader on the application of new technologies for positive social change. He currently serves as Director of Social Innovation at the Qatar Foundation’s Computing Research Institute (QCRI). Patrick also authors the widely respected iRevolution blog and tweets at @patrickmeier.

Wreck at Coleman's Station, New York & Harlem R. R., March 13, 1888. Photo courtesy NOAA Photo Library.

I’ve just spent a riveting week myself at the 2012 National Geographic Explorers Symposium in Washington DC, the birthplace of the National Geographic Society. I was truly honored to be recognized as a 2012 Emerging Explorer along with such an amazing and accomplished cadre of explorers. So it was with excitement that I began reading up on the history of this unique institution whilst on my flight to Doha following the Symposium.

I’ve been tagged as the “Crisis Mapper” of the Emerging Explorers Class of 2012. So imagine my astonishment when I  began discovering that National Geographic had a long history of covering and mapping natural disasters, humanitarian crises and wars starting from the very first issue of the magazine in 1888. And when World War I broke out:

“Readers opened their August 1914 edition of the magazine to find an up-to-date map of ‘The New Balkan States and Central Europe’ that allowed them to follow the developments of the war. Large maps of the fighting fronts continued to be published throughout the conflict […]” (NG 2003).

Map of ‘The New Balkan States and Central Europe’ from the August 1914 "National Geographic Magazine." Image courtesy NGS.

National Geographic even established a News Service Bureau to provide bulletins on the geographic aspects of the war for the nation’s newspapers. As the respected war strategist Carl von Clausewitz noted half-a-century before the launch of Geographic, “geography and the character of the ground bear a close and ever present relation to warfare, . . . both as to its course and to its planning and exploitation.”

“When World War II came, the Geographic opened its vast files of photographs, more than 300,000 at that time, to the armed forces. By matching prewar aerial photographs against wartime ones, analysts detected camouflage and gathered intelligence” (NG 2003).

During the 1960s, National Geographic “did not shrink from covering the war in Vietnam.” Staff writers and photographers captured all aspects of the war from “Saigon to the Mekong Delta to villages and rice fields.” In the years and decades that followed, Geographic continued to capture unfolding crises, from occupied Palestine and Apartheid South Africa to war-torn Afghanistan and the drought-striven Sahel of Africa.

The iconic cover featuring a young Afghan refugee from June 1985. Photo courtesy NGS.

Geographic also covered the tragedy of the Chernobyl nuclear disaster and the dramatic eruption of Mount Saint Helens. The gripping account of the latter would in fact become the most popular article in all of National Geographic history.

Today, “new technologies–remote sensing, lasers, computer graphics, x-rays and CT scans–allow National Geographic to picture the world in new ways.” This is equally true of maps. “Since the first map was published in the magazine in 1888, maps  have been an integral component of many magazine articles, books and television programs […]. Originally drafted by hand on large projections, today’s maps are created by state-of-the art computers to map everything from the Grand Canyon to the outer reaches of the universe” (NG 2003). And crises.

“Pick up a newspaper and every single day you’ll see how geography plays a dominant role in giving a third dimension to life,” wrote Gil Grosvenor, the former Editor in Chief of National Geographic (NG 2003). And as we know only too well, many of the headlines in today’s newspapers relay stories of crises the world over. National Geographic has a tremendous opportunity to shed a third dimension on emerging crises around the globe using new live mapping technologies. Indeed, to map the world is to know it, and to map the world live is to change it live before it’s too late. The next post in this series will illustrate why with an example from the 2010 Haiti Earthquake.

Patrick Meier is a 2012 National Geographic Emerging Explorer. He is an internationally recognized thought leader on the application of new technologies for positive social change. He currently serves as Director of Social Innovation at the Qatar Foundation’s Computing Research Institute (QCRI). Patrick also authors the widely respected iRevolution blog and tweets at @patrickmeier.