The Navy let me hunt Chinese warships at 16

The job was simple to describe. Look at a picture of the ocean taken from very far above, find the ships, and say what kind of ships they are.

That’s the whole task. A gray sliver on a gray sea, a handful of pixels long, and a question: is that a fishing trawler, a container ship, or a People’s Liberation Army Navy frigate?

Humans can do this. Analysts do it all day. But humans are slow, the oceans are enormous, and the cameras never stop. So the assignment was to make software do it: machine learning that identifies ship types from overhead imagery, Chinese naval vessels very much included.

Ships, from above, at a distance

The lab I was in already had software for part of this. It was called RAPIER, and it could detect and classify ships in satellite imagery. The gap was video: the full-motion-video version could find ships in a live feed, but it couldn’t tell you what they were. Detection without classification. There’s a ship, good luck.

My summer lived in that gap. I trained convolutional neural networks to identify ships in full-motion video, using Caffe and Nvidia’s DIGITS.¹ This was 2015, when “deep learning” was just crossing over from research papers into rooms like that one, and a big part of the job was simply getting the pipeline to run: labeled ships in, trained classifier out, over and over.

The other half of the work went the opposite direction: instead of flattening ships into overhead signatures, rebuilding them in three dimensions. From directly overhead, most ships look like the same gray rectangle, and the same ship looks different from every angle. So I wrote a C++ desktop application that reconstructs 3D models of ships from imagery using structure from motion: feature matching, bundle adjustment, scene reconstruction. Frames in, rotatable geometry out. The goal was what my project poster called omnidirectional classification: know the shape, and you can recognize the ship from anywhere.

I named the program C3PO.² The poster’s subtitle, under two renders of the same vessel from different angles, was “They’re the same ship!”

The actual poster, rendered from the PowerPoint that still sits in the internships folder. Top right, two photos of the same vessel and the whole problem in four words: “They’re the same ship!” Bottom left, the payoff: frames of a littoral combat ship in, rotatable point cloud out.

One August afternoon that summer: my laptop grinding through exhaustive pairwise feature matching, photographed the way you photograph something you can’t quite believe is working. The file paths still say BenClark.

Okay, where was this

Not at a startup, and not at school.

This was SPAWAR Systems Center Pacific: the Navy’s research lab in San Diego, in a group called the Advanced Imagery Analysis Lab. The place with badges and escorts and buildings you don’t wander into. The Office of Naval Research runs a program called SEAP that places students in Navy labs for eight paid weeks, I won one of the spots, and from June 28th to August 14th, 2015, that lab was my job. Forty-five hours a week.³

Which brings me to the other detail I’ve been holding back. I was sixteen.

Sixteen, with a backpack, showing up at a Navy lab every day to teach computers to recognize warships. The badge photo is a child.⁴ And I don’t have to describe the poster session, because the Navy photographed it:

SSC Pacific’s own newsletter, covering the end-of-summer poster session. That’s me in the tie, the C3PO poster on the screen, and the Navy’s caption doing the introductions: “An ONR summer intern discusses his work on the Rapid Image Exploitation Resource (RAPIER) program with Lee Zimmerman.”

I should credit the person who made the summer real work instead of babysitting: my mentor, Dr. Katie Rainey, who handed a sixteen-year-old an actual technology gap and treated the results like they mattered.

And nobody at the lab treated the work as a toy because a teenager was doing it. The model either classified the ship or it didn’t.⁵