How Forensic Architecture Uses 3D Reconstruction to Investigate Human Rights Violations and State Violence

The success of open source investigations often hinges on the availability of visual evidence like photos or videos found on social media, CCTV footage, or satellite imagery. But what can you do when that evidence isn't conclusive or simply not available? 

The research group Forensic Architecture had to answer this exact question in one of their most recent investigations this year in May into a 9-year old police shooting case in the U.K. "There wasn't any direct visual evidence of the moment that we were most interested in," says Nicholas Masterton, who works as a researcher for the group. 

Forensic Architecture, which is based at the University of London and uses architectural techniques and technologies for its investigations, had to rely on written testimony and a shaky video that only showed the aftermath of the incident. By reconstructing the event in three-dimensional space in meticulous detail, they were able to shed new light on a case that happened almost a decade ago. 

"I'd say one of the core techniques for Forensic Architecture is that we would have multiple perspectives or multiple aspects on a single event," says Masterton, adding that two-dimensional footage only gives investigators partial information about where people or things were in space at a fixed moment in time. 

The sort of meticulous 3D reconstruction used in investigating the shooting is something Forensic Architecture has pioneered and used in dozens of investigations. 

Earlier this year, Forensic Architecture investigated a devastating explosion in Beirut's port by collecting videos, photos, and other documents to construct a timeline and a precise 3D model of the incident. Together with the New York Times' visual investigations team, the group looked into the Syrian government's use of chemical weapons using 3D-modelling software to digitally reconstruct attack sites and the gas canisters used. 

The group has also published research on the Grenfell Tower fire in London, the murder of Halit Yozgat by a German neo-Nazi group, and a factory fire that killed 260 people in Pakistan. Another project by Forensic Architecture used photorealistic 3D renderings to train machine-learning algorithms to identify munitions and weapons used in conflict zones.

A police shooting 

In an investigation published in The Guardian and on its website this year, Forensic Architecture looked into a 2011 police shooting of Mark Duggan, a 29-year old Black man, a highly contentious case that sparked the biggest riots in the country's modern history. 

Officers of a gun-control unit had followed Duggan as he traveled in a minicab after reportedly picking up a gun. They forced the minicab to pull over, and Duggan jumped out as the vehicle came to a stop. Within seconds, an advancing officer fired two shots. One passed through Duggan's arm and struck a second officer, while the second fatally hit Duggan.

The officer later told investigators that he saw a gun in Duggan's hand, but following the shooting, that gun was found several meters away from where Duggan had been shot, on a patch of grass. None of the officers reported that they saw Duggan throw the weapon.

The U.K. police watchdog’s investigation into the incident, published in 2015, concluded that the “most plausible” explanation for the gun’s location was that Duggan had been “in the process” of throwing it as he was shot.

Whether Duggan was holding the gun was crucial to the Forensic Architecture investigation.

"We did this investigation to find out what was really taking place [...] It's a very short space of time, probably 5 to 10 seconds, where this person is exiting a minicab being shot twice," Masterton says. "And then, there's this thing where a gun is found in the grass field next to the site, and the question of [...] how did that gun get to the grass? Was it thrown by him as he left the minicab? Did the police see the gun?"

Forensic Architecture didn't have a lot of visual material to work with and instead relied more on written documents such as transcripts and statements from police officers. Masterton and his colleagues sifted through hundreds of pages of documents searching for details that could be useful for 3D reconstruction; a process Forensic Architecture calls data mining. 

The team read through the officers' testimony to reconstruct the timeline of the shooting and the individual officers’ movements around the scene. 

"So we do that for one officer. And then we do that for another officer. And we just kind of keep going through and make a spreadsheet of all of the events that they witness," he says. "And that spreadsheet can be converted into a set of moments in a 3D model."

Another way they gathered data was by analyzing a video shot from a nearby building that showed the movements of the police officers around the site immediately after the shooting. "That was very interesting because the quality of the footage was such that we couldn't tell the identity of everyone in the frame. But due to what we knew about things that they had said, we knew that certain pixels, like muddy little blocks of pixels, were certain people, and we could work out some of their interactions and some of their movements on the site," Masterton says.

The researchers also traveled to the shooting site in Tottenham, in North London, and captured photographs to later turn into a 3D environment, in a process called photogrammetry. The group used motion-capture technology to understand whether Duggan could have thrown away the gun before he was shot. They also gathered biometrical data to make the resulting 3D model and animations as accurate as possible.

In the end, Forensic Architecture’s models found that the police’s view of events -- that Duggan most likely was in the process of throwing the gun as he was shot -- was implausible.

Finding the right method

Masterton says that no two projects are alike and his group has to come up with unique methods for each of their investigations. 

“You don't know the landscape of the project that you're going to work on until you've arrived at it. So often, you find that there's a technique, which you think will work for something, but actually, it doesn't.”

A lot of the software and techniques used to reconstruct scenes in 3D are designed for ideal conditions with pristine data. But this doesn’t match the reality of a lot of user-generated content, which is often shaky or filmed at a relatively low resolution. “Often we find ourselves working with tools that are designed for, you know, a Hollywood-like really nice, carefully planned shot, when actually we're dealing with something that's a completely different type of content.”

These problems mean that the researchers spend a lot of time trying to figure out the quality of their available data and working out the best technique for a certain type of footage.

“In a way, a lot of the work is about overcoming those limitations, or looking at the same limitation, but from a different angle. And then you learn something new about it,” he says. “If you zoom in far enough, in the end, you're always going to be looking at two pixels.”