Here’s my final presentation on trying to put sound and text together. Everything’s up for discussion, think I may have found a direction to go in.

I’ll present this in class so no need to read now, but here’s the link to the presentation.

Why seeing the un-seeable should be a superpower available to everyone.

For the last few years Arlene Ducao, a researcher at MIT, and her design partner Ilias Koen, have been building an awesome piece of software. It’s a visualization tool with the uncanny power to reveal hidden worlds; its called Open IR and they’re about to show me their first Prototype.

I meet Ducao in her office on the Media Lab’s 3^rd floor. It has a narrow view of the Boston skyline and what look like drawing boards are leaning sideways against the desks. Prototypes for other projects are scattered everywhere. Ducao is petite, colorfully dressed and bursting with enthusiasm. She greets me with a huge grin.

Her partner Koen is already here, but invisibly so because he’s only with us via Skype. His avatar on Ducao’s IPad is a comic book hero I don’t recognize but he assures me he really is in Brooklyn. We position him on my knee so that he can see Ducao’s large screen monitor where the demo is about to happen. Every now and then the noise of New York traffic breaks the studious peace of the Media Lab.

The first thing Ducao loads is a Google Map satellite view of Jakarta, Indonesia’s capital city on the North West coast of Java. It’s the largest city in South East Asia; home to over 10 million people. The map displays all the usual features of a sprawling metropolis as seen through Google-vision; a digital network of yellow lines and white text with the names of towns, suburbs and major roads imposed on the dusty green fuzz of reality.

According to Ducao, what we have here is an optical satellite image that has high spatial resolution (each pixel represents one meter) and lots of street names and landmarks but not much about the natural world. “You can just make out what’s vegetation and what’s urban sprawl, but it’s all a bit murky”.

Then she switches to the Open IR interface, and everything changes. With a similarly scaled satellite view of Jakarta, she begins to transform the city before our eyes. Each new click reveals a new layer of reality and features that were barely visible to the naked eye are suddenly amplified with eye-popping colors.

With the first click, the city itself turns a shocking pink which leaks down into the southern hillsides in spidery clotted lines like spilt nail polish. The pink areas are all the impermeable surfaces, the built world of concrete and tarmac.

Jakarta's impermeable surfaces in hot pink.

With the next click, the colors switch. The metropolis turns green but it’s now spattered with a livid red that gradually blooms at the cities edges into a red cloud billowing out over the southern slopes and the land to the east and west. Where red meets green the border is ragged, the colors interlace like fingers meshed. From the IPad on my knee Koen tells me through traffic, that the red is everything that is photo-synthesizing.

With the final click, the ordinary-looking fields to the North East of the city give up a secret. They throb a deep navy, almost black. This view highlights moisture and the fact that they’ve turned the same color as the sea means they are very wet. Paddy fields perhaps? “Yes that’s what we think, either that or marshland” says Ducao. A close zoom in with Google maps view confirms they are in fact rice paddies.

What we’re seeing here is our familiar world enhanced by infrared vision. Its the superior machine-vision of Landsat 7, a satellite that’s been part of NASA’s long-running Earth Observation System since 1999 and is orbiting somewhere above us right now.

Ducao and Koen discovered this rich dataset when they were working as science visualisation artists at the New York Museum of Natural History. Previously they’d both trained as computer artists at the New York School of Visual Arts.

“Coming out of there, most people wanted to work for Pixar or wanted to become gallery artists but I wasn’t feeling either of them,.” Ducao says, laughing.

Instead they ended up making art out of science for the Museum. “Our job was to keep the permanent exhibitions fresh,” says Ducao, which they did with a mix of video and computer installations. They worked in small teams; a producer, an animator and a scientist, with the scientist doing most of the data processing.

“Gradually we started doing more and more of the data processing and visualisation ourselves,” says Ducao. “When we came across the Landsat data and we had some sense of its power.”

NASA describes Landsat 7 as not only providing the longest record of the earth’s continental surface as seen from space, but a record “unmatched in quality, detail, coverage, and value.”

Unlike the optical satellite images we are familiar with, Landsat’s specialty is multispectral imaging, in other words it measures energy reflected from the earth’s surface across a broad range of the light spectrum from the visible to the lower frequency mid-infrared.

The ability to ‘see beyond the visible’ has given scientists an enhanced picture of what goes on on the ground in places they can’t easily get to, particularly environmental changes. They’ve used it to analyse oil spills, earthquake damage, logging-related landslides, soil composition, fault lines and water-depth. In each of these situations, objects on the ground consisting of organic and inorganic matter reflect back a unique array of electromagnetic waves according to their chemical composition and temperature. The more of these waves that can be detected, the more scientists can determine what the objects are made of, which is why multispectral measuring devices are considered so useful.

Since 2008 the data has been available for anyone to use. And indeed NASA publishes a Science Data User’s Handbook. The problem is it comes in what Ducao refers to knowingly as “g-zipped text files”; reams of encoded data that need to be converted into something recognizable before they can be interpreted – a time-consuming process. “That’s why its really only scientists who use it” says Ducao.

The pair believe that this data deserves to be liberated. Open IR is all about releasing it into the wild.

The first thing they did was to create an animation using data about China’s Pearl River Delta, on the mainland opposite Hong Kong. They called it the Human Footprint because it focuses on human impact on the natural landscape. But the animation also demonstrates powerfully how different combinations of visible and infrared light can be used to reveal differences in land cover such as whether a field is ready for planting, or for harvesting, or what’s recently be deforested or built.

As Ducau explains, its not that IR shows you things you can’t see, its more that it detects more detail and enhances what is rather murky with merely optical vision.

It was the Japanese Tsunami that sharpened Ducao and Koen’s thinking about how the data might be used. As a wealthy country Japan had access to the best quality map resources to guide its disaster relief work, in contrast to the developing world and the enduring chaos that remains in many areas hit by the 2004 Tsunami in South East Asia. Ducao and Koen realized that easily accessible online IR maps could help the most vulnerable understand changes to their environments and vastly improve their disaster relief efforts.

Jakarta was a natural choice for their next visualization. Its not only economically and environmentally vulnerable but some NGOs, like Open Streetmap are already working with local communities to improve their woeful ground-mapping resources so it was easy for Ducao and Koen to find partners to work with.

Japan’s crisis supplied another idea – the risk map, created by experts to determine evacuation zones and disseminated to the public via the media.

When Ducao clicks on the risk-map they made for Jakarta, a grim picture appears. The whole city turns ‘at-risk’ red, including areas far from the coast. The low-lying rice paddies are, of course, set to be obliterated, but Koen is quick to point out that the risk map is not predictive. It merely reflects how vulnerable each land-cover type is to flooding, based on a ten-point list gleaned from previous crisis management research.

Jakarta Tsunami risk map.

There aren’t really any surprises that come out of this list (low elevation concrete is the most vulnerable, high elevation vegetation is the least), but the risk map certainly makes it clear that smooth hard streets let water flow much more easily than barriers of vegetation, even if they are at greater elevation.

Ducao and Koen are looking for partners to develop new projects with Open IR. They see a plethora of users amongst the lay-public, from agricultural workers to architects, urban planners to environmental activists, public health workers to community groups and even travelers looking for the greenest city to visit.

Perhaps the most compelling use of Open IR will always be for crisis management in the aftermath of a natural disaster.

“Imagine if you could combine this environmental data, with a crowd-map like Ushahidi” says Ducao, “then you’d have location-based human information and dynamic real-time environmental information all in one place.”

But for that dream to happen, they’re going to have to jump ship. Sadly Landsat 7 suffered a hardware component failure in 2003. Its still up there measuring, but bits of information will always be missing.

Koen tells me they now have their eyes set on ASTER, a Japanese sensor on board NASA’s Terra earth orbit satellite (also launched 1999 ). Its capable of recording energy from 15 different bands of the electromagnetic spectrum and has double the spatial resolution of Landsat allowing you to zoom in closer.

“We are excited because you can put in a request to have the satellite collect some data for you” says Koen. They’ve put in a request to update their Jakarta visualisation that’s still pending.

For now, Ducao and Koen are building more visualizations city by city. They are currently working on London (although no risk map yet)  and New York, and will be building up a database of cities and ecosystems at high risk either of natural disaster or climate change.

But its not all disasters and emergencies, Ducao and Koen are working with a local community organization in Brooklyn that is trying to reclaim vacant lots for public use by turning them into green spaces. 596 Acres have retrieved a complete list of vacant lots from local government and are surveying them, lot by lot, to see what’s there. The New York Open IR map will automate that process, revealing which of the lots are already green and could be developed further, and which neighborhoods are in dire need of more green space, thus helping the organisation focus its resources.

Infrared visualizations may look esoteric compared to the average google map, more like an abstract Rothko painting, but with Open IR’s democratising vision, to release the data and make sense of it, we may soon all be able to understand their bright colors and strange visions and many will be using them for social change.

“My best friend recently said to me,” says Ducao, “that climate change is getting personal. It used to be something we’d only read about or watched on the news, but now we’re experiencing it firsthand.” If that’s really how people feel, then having environmental data visible in our online maps may prove a really useful tool and who knows what people will do with them.

NASA’s satellites maybe old, but as is happening in so many areas of culture, it seems the participatory internet is about to give their data a new lease of life.

Challenge: can I convince the class that a soundtrack might enhance their future journalism?

My Approach

In our last class I was really impressed with how you all approach problems, interrogate ideas and map out paths to a solution so clearly. I’m not a linear thinker. Here’s the proof – my plans for my year at MIT, scrawled back in August last year:

But inspired by the systems thinkers – I’ve tried to be more linear. And if anyone has suggestions for a clearer path, do please get in touch:

PATH 1 – WEEK 1

• Talk to a couple of DJs I met last semester who live-mix sound effects into tunes. Ask about sound apps/software, ideas, examples etc.
• Choose an already written news story that I can imagine a sound track to.
• Find free sound effect archives or sound sharing sites.
• Work out how to scrape sound from internet sites (Sound Cloud, You Tube etc).
• Work out how to edit this sound to fit the story.
• Work out how to most elegantly embed the sound on the story page (using Sound Cloud).
• Try it out on someone for feedback.
• Assess the workflow for improvements and questions.

PATH 2 – WEEK 2

• Write a story myself. (I’m currently writing about suicide, a serious subject that might not seem ideal material for a soundtrack, but because of that, I feel it actually offers an interesting challenge to the sound track idea.)
• Gather sound effects with microphone myself (try various microphones – cheap dictaphone, pro mic and smartphone)
• Edit sound track.
• Upload sound to story.
• Try it out on someone for feedback.
• Assess workflow.

Questions I’m asking myself:

What kind of stories would really benefit from sound?

How would a soundtrack change the kind of stories I might want to report on and how I write them?

How complicated are the tools I’m using, how could they be simplified?