Giant Manta, Meet Tiny BoxLatest update November 11, 2018 Started on July 18, 2018
We are attempting to squeeze a 20 ft wide Giant Manta into a tiny box.
We plan on capturing natural history footage of Giant Mantas (Mobula Birostris) with a custom 32-camera rig built specifically for hologram display and ultimately bring back Mantas in a small holographic device that can sit on your desk. What can go wrong?
We debriefed in front of everyone
After we got back to the U.S. Alex whipped together some cuts of the Manta Hologram Video.
On Nov 8th, I spoke about this project at the All Hands On Deck conference @ MIT Media Lab.
Here is a sample of the Holographic video below:
Heading back to the states
When we reached land last night, after our victorious dive, Alex quickly began processing our dive footage.
I waited to hear back on whether or not we got what we needed. Some hours went by, I was too wiped out to be concerned. I woke up sometime later and made it back to the dive shop and onto wifi. Got a message from Alex via whatsapp. "We got it".
I called to be sure. As far as he could tell, all cameras had stayed on the entire time and we have what we need to create some Holographic content.
We got a chance to celebrate and toast at a small restaurant in the area. Success, as far we were concerned, tasted really really good.
Back to the states.
This is what, Lucini pourin' from the sky
Today was the worst day and the best day all wrapped in one; Our Last day.
Bottom line is: We got it. We think we got it. Everything lined up. It came together. There's more to the story below.
Enchanted by Mantas
Everyday that we've been out here, we've had two dives , two opportunities to capture what it is that we are looking for; Stable footage, at the right distance from the mantas, in waters that provide 30-40 feet visibility.
Dive 1 -
I took over the camera rig and jumped into our dive ready to get what we came here for. I checked every camera and they were all good at the 30ft rest stop. We pressed on with a dedicated dive guide that would take us to our own location apart from the rest of the science team.
We quickly spotted a very large manta below us that looked like it was standing still. It was the perfect subject for a shot that requires stability and not having to chase a manta with a rig. Once you get into a chase it's over. You're going to burn lots of oxygen trying to keep up, you really need the manta to be ok with you being around it, so it's moving slowly and with curiosity in you as well.
Seeing my opportunity I dove deeper, keeping my eye on my depth gauge. I knew I was pushing the limit of where we thought the cameras might begin to shut off.
But I was close and I checked all the cameras quickly for red flashing record signals, and they are all firing. As I approached the water became both colder but also much clearer and the manta was not moving but gliding in place.
Not only did I think i had finally gotten the footage we needed but I decided a head on view was better than overhead, so i descended just a bit more to be at eye level with this manta - I look at my depth gauge i'm at 90ft. Mask palm.
Alex descended down and took a look at the cameras from above and started pointing up but I was fixated on the manta and kept chasing.
Sure enough we get back up to the boat and many of the cameras are off. Alex is looking dejected. Face in palms he looks up and says, I was trying to tell you you were too deep.
Damn. We have one dive left on this trip and i'm not confident about ANYTHING we've picked up, up to this point. MAYBE the turtles, but that's not what we're here for.
Dive 2 - Seeing Alex in a frustrated state, I slap him on the back for the last hoorah. We got this, Alex. The only thing I had left in me, was to be overly optimistic. I felt good about all the different variables; clear water, mantas present, and remembering to stay above 50 feet.
Sun would have been nice (every day, every hour has been overcast on this trip) but we got the ideal conditions to make this work.
In typical life magic fashion, the clouds part and the sun rays break through completely transforming the feeling on the boat. Now I was definitely adamant that we would get what we came for, and yes it's our last dive on our last day but that's how things work.
With the rays beating down on the boat that energy is back up and we make some adjustments, this time we let the captain know that we need to be taken somewhere where we can find mantas at a relatively shallow depth. The gamble is that the water clarity would be comprised at that depth, but hey at this point, I'd rather have a blurry manta than no mantas in that box.
Upon diving we are guided to 30 ft deep plateau where Mantas are known to circle and come in to be cleaner. and we wait. and we wait some more. Because we're at shallow depth, we have some time on our side, as our tanks our lasting a bit longer than our previous dives. At minute 10 into the dive we see some mantas circling the outskirts of where we've planted. I definitely have an urge to go out an chase them because all the conditions are right, but they are deceptively far, it would be very easy to go out and chase them and come back with absolutely nothing.
I stay put and wait. Now they are coming in closer and feeling confident to swoop in for some cleaning. Now it's on - the sun rays are brightening the ocean, our depth is on average around 25 ft, all cameras are firing and i'm seeing some slow moving mantas moving my way.
I kick up to get eye level and realize we are at what can only be described as a manta airport, one after another, waiting in line, the mantas swoop in slowly to be cleaned and all of our 32 cameras are capturing all the action. The water visibility is what you could ask for given all the variables and one after another till the end of the dive, mantas are showing up.
The last dive, of our last day on this expedition provided all the footage we needed to walk into post-production confidently.
It was the depth, as far as we can tell, that caused the cameras to turn off throughout this trip. Apparently these offbrand gopro housing were bowing at relatively shallow depths of around 60ft (not the 150 ft they claimed) We never went below 30 ft on this dive and they all remained on.
Ecstatic is an understatement. When we got to the surface we could hear everyone at the surface cheering because of how incredible the experience was, all around.
Photo attached says it all. It came together.
Also, that boat ride back was a great opportunity to reflect on this crazy project.
Back at it
We think we have this figured out. We've narrowed the issue of cameras being turned off underwater.
On a prior trip we were able to get footage of turtles at the surface with all cameras rolling perfectly. But on a more recent trip, when he handed the rig to two of our dive guides they said they noticed the cameras began to shut off around 60ft.
After some research Alex located a possible cause; it could be the SD cards we are using. The brand / make we chose has had some known issues, when it records past a certain amount of time. If this is the case, we are going to have to face the idea that we won't be able to come back with all the footage we need, because finding micro SD cards in this small town is not happening.
Another possible issue could be the underwater housings. GoPro doesn't make housing for these gopro sessions. So Alex bought what he thought were some housings that had some pretty decent reviews.
It could be the the housings are bowing and pressing on the buttons to reset the cameras at a certain depth. We noticed that some of the cameras are coming back not turned off, but switched to timelapse mode, even though we were very careful not to touch cameras once they were set.
Personally, i'm hoping its the depth issue which we can work around much easier than a set of damaged micro SD cards.
Each one of these dives does require alot of energy, so writing this update tonight is a struggle especially since things are lining up exactly right.
Even if we get all the tech right we still have to contend with wildlife showing up when you want it to, under the right conditions. It was rocky out there today and currents were strong, so navigating with this rig became very challenging. On today's second dive we just left the rig on the boat after it had been carefully prepped and turned on for us. It was just too turbulent and we needed to get under the surface almost as soon as we made it to the front of the boat.
Another concern is Alex has to leave tuesday morning, we are running out of opportunities. The good news is there have been mantas every day and dive we've gone out, we just need to figure out the camera issue.
Quick Detour to Present at the University of Guayaquil
We took a day off the water to head into the interior and present at the University of Guayaquil.
Alex talked a bit about the vision for a Holographic aquarium of the future. And he did it completely in Spanish (He's ok in Spanish), because Alex does what Alex wants.
I queued slides in the back to make sure he was supported by beautiful visuals. Overall, it turned out well. One thing we learned for sure is university students and faculty alike are definitely drawn to the idea of Holograms.
There was a group gathered over the display table to see some other holographic experiences that Alex loaded up prior to our trip.
Narrowing it down
We are having amazing dives with amazing mantas, but we cannot figure out what is going on with the rig.
Basically, we need all cameras to turn on and stay on throughout all the dives. We've tried turning them all on while we are underwater, and they work, but when we get to the boat we find them some of them have turned off.
If we cannot get all cameras to stay on during our dives, we'll need to figure how to deal with the lapse in footage or try and troubleshoot this. We are about half way through our expedition and although we have had some amazing footage, we don't have what we came here for which is stable footage of a very large manta ray at the right distance with the right water clarity.
The upside is we are learning about how the rig could be built in the future, there are some turning issues that we may be able to solve by adding small thrusters, but that's a whole other discussion.
We've discussed some potential causes to the cameras going off, but no consensus yet.
* First part of this Mission Accomplished*
Our final night out with some of the team members and we're feeling really good about how this expedition ended. Having worked on several scripted TV shows, I can only say this is life imitating art. I do not know why everything came together on that last dive. Maybe the pressure helped us focus and make the right calls (that's my vote) or maybe it was just luck, but it all came together.
Alex checks in and let's me know, we got it. He did some scrubbing through all the cameras and can say confidently that we have what we need to create some Holographic content.
We're heading back to the states with some post-production ahead of us, but also feeling like success is already here.
What had happened was
We've been out on the water a few days. Water is amazing, mantas are here.
So much to say here, but when we get back to land we have tons of post to process, and Alex is leading the charge here so really he is all tied up till late night, every night.
There are definitely giant mantas that welcomed us and amazing display of a healthy ecosystem here - wish everyone could see this. Will post some photos to this post of some great whale breaching, mama teaching calf some breaching styles. Today's lesson was straight up belly flop and the classic 180 twist.
On the camera rig, it's not super easy to maneuver with, but luckily if mantas like you they stick around for a while we had some nice moments, but when we got back up to the surface some of the cameras keep shutting off. Trying to figure this out.
Lots of testing to be done. Will update soon.
Lock and Load
The night before we head out on our first day:
We arrive at Puerto Lopez where we will be based for the next 10 days.
We're excited to get going but there are tons of unknowns to figure out before we actually get out on the boat.
Puerto Lopez is also a nice, low key place to figure all this out.
So first things, first. We need to get all the cameras built, load SD cards, put all batteries to charge, label all cameras and make sure we know that each card was place in the correct camera (Unless you want post-production to be a nightmare. Freddy on Elm Street status.)
AND EVERYONE PITCHED IN. Age is just a number, right?
After a few hours, everything was prepped and we're feeling good going into the following day.
Felt like Christmas.
Spirit Airline is amazing.
We read somewhere that posting contrarian views as a headline, immediately jumps readership by 45%.
Carlos: Alex is a G. You can decide what kind of G that is. He managed to squeeze the 8ft rig, all the accessories (and there are a lot. ) and 32 cameras into a tight package. Boston George would be proud.
"We" decided the 12ft rig was definitely a no go (and by we, I mean I kept hinting to Alex that there is no way we are taking that thing with us.)
I decided to try and save on my flight by booking with Spirit, so i'll be connecting to about 7 different cities before I make it to Ecuador, and Alex and familia will be arriving about 4 hours before me. But I saved a lot more than him on my flight. Until I had to pay a fee for my hat. And shoes. J/K. But really though, my flight ended up being a lot more expensive since I had to take along the cameras and some new BCD's we just picked up from OCEANIC.
See you in Guayaquil, Alex.
Streamline me or I walk...I mean it.
Alex: No matter how streamlined I could make the design, when you multiply the headaches of one camera by thirty-two, they can easily get out of hand. I’d have to keep the cameras charged, get the video files onto a computer, keep them clean and keep them waterproof. In my first tests, it took me about eight hours of post-processing to get thirty seconds of footage with this camera. None of it was particularly hard, it just takes a long time to deal with this amount of cameras, and it’s a ton of data — every minute is 7.5 gigabytes of footage with this rig. For a ~45 minute dive, I’d be pulling 330 gigabytes of footage off the cameras. I gulped a bit, bought two 4TB hard drives, and put them in my bag.
Open Waters Test
Carlos: I've been clearing my plate of other projects, so I can focus on this expedition and in the meantime, Alex was taking the camera out to test in a pool environment (read: safe, predictable environment), from prior post.
Alex was excited about how the tests turned out, but I had my reservations as to how that would translate into an open water scenario, but it was a good sign that at the very least, the build operated the way it was meant to, except for the losing a camera thing.
As a result of that test, we can get a sense of how much drag the rig creates and the underwater housings didn't flood. So all good things.
But I wanted to see this thing operate in some real conditions. So what better place to go then Newport, Rhode Island, where the visibility is about 2 feet on this special day.
What did we find from our open water test?
All cameras fired upon turning them on and it was easy to navigate across all 32 cameras to get them going without a sync'd ignition, but we'll take that.
An important part of what we learned is maneuvering this rig with two people is not going to be easy, and also pool floats to have the rig float at a certain depth was messy, as the line got tangled around our scuba tanks several times.
We need to make some adjustment to the design, like adding handles to the ends of the rig, so it can be coordinated between two people much easier. Also, I think we have to nix this pool noodle idea. Isla De La Plata will not be as kind, as it's truly open water (as opposed to the bay we were in today.)
Also, Mantas can become tangled in the lines that we are hanging from the pool noodle to stabilize the rig, so it's definitely out. We'll need to adjust to make this rig completely supported by us.
(No lightfield video from this test due to the phenomenal clarity of Rhode Island’s waters — it would make everyone else too jealous)
After these tests, we figured that we had a seaworthy camera. Still, there were a few last details to work out.
Let's test this thing.
Alex: To test underwater, I enlisted the help of my local dive shop Dive On It, who got me into a pool and helped me set up a camera for my first underwater tests.
This was still far from the mobile camera rig that we would eventually need for the mantas, but I wanted to see if there were any significant differences between imagery captured underwater and above water. I started to get a feel for how to frame an underwater shot, and also came to the conclusion that, yeah, twelve feet is too damn long of a camera to dive with, and shortened it to eight feet. I also lost one gopro due to a faulty seal on one of the doors.
The good news is I didn't need floaties:
It's time to refine this and move towards a more realistic environment.
What’s an underwater holographic camera and how do you use it?
Foreword by Carlos: When it comes to Hologram camera construction and considerations for post-production, I listen to Alex explain how it works and nod. Every nod reinforces my trust in Alex, because otherwise I have no idea whether what he is telling me is even remotely going to work. I'm not saying it's particularly difficult to understand, I'm just saying there is no Alex 101, it's just Alex AP. As someone driven to communicate compelling stories, my question is always first and foremost, the Why. But below you'll get a good idea of the How.
So this post gives you an opportunity to sit in my seat and nod, politely.
Alex: It took us a while to design this holographic camera and learn to film with it. The word hologram gets thrown around a lot these days, and we’re using it as a shorthand for a nerdier term, a lightfield. Lightfields are the big, abstract idea behind the original 1940's-era holograms and the work we do now at Looking Glass — it’s all about capturing all the available information about the rays of light in a scene. When you capture the lightfield of a scene, you can work out what the scene looks like from different angles, at different planes of focus and different apertures. Interestingly, you can do all this after you take the footage, using a bit of computation, which opens up all sorts of new possibilities for photography and videography.
The lightfield camera that I built is tailored to our displays, which work by sending many views of a 3D scene out into the real world. If I want to present the illusion of a 3D manta ray in a hologram, I need to capture 32 images, simultaneously, from 32 different positions. Our Looking Glass display sends out those images on one axis, so your left eye sees one image and your right eye sees another. To capture these views, I laid out the cameras in a long, straight line.
There are lots of people in the world who make lightfield cameras, and basically all those cameras are really fancy, high performance cameras that take up entire rooms and don’t have a prayer of going into into the world, let alone under the sea. To make a compact, self-contained system, I had to dramatically simplify the camera design. so I went with a philosophy that we use a lot in my company: use simple, imperfect hardware and make up for the imperfections with software.
About that — I wrote a lot of custom software to make this camera work. Every frame, the camera captures sixty-six million pixels, and to get the lightfield to send over to the display, we have to take all that information and distill out the paths and colors of four million rays of light. I pulled a few all-nighters in the lab, fueled by coffee and the rapidly approaching date on my plane ticket. There isn’t a wealth of lightfield video editing software out there, so I focused on making software that would take care of the esoteric lightfield processing and give me regular old video, packed with unusual information, that I could edit in Premiere. I ended up with a kind of Rube Goldberg software machine, linking together a handful of different programs, but that didn’t bother me — this was a prototype camera, and that’s how it works: you make a prototype, you test it out by filming mantas in Ecuador, and then you refine it, man.
I knew that I wanted to use all off-the-shelf components to simplify the camera design. I settled on Go-Pro cameras because they are small, relatively inexpensive, and have a bunch of readily available mounts, dive housings, and they had a good rep among divers. The Session cameras in particular interested me because they are extremely small and don’t have any other battery-eating components like screens — it’s basically just one button to start and stop recording to an SD card, and that’s all you get. I put a fancier gopro with a viewfinder at either end of the line so we could frame the shot, and I called it a camera.
The small cameras came in handy for a practical reason, too — the lengths of the camera can really add up, and we were wary of moving a giant honking camera around, seventy-five feet below the waves. The small session cameras, with dive housings, had a spacing of 2.75 inches from center to center, giving me a total length of just under eight feet. I built the whole frame out of aluminum extrusion and stainless steel bolts that wouldn’t rust underwater, and made it so the whole beast would break down into two foot lengths and fit into a standard suitcase for travel.
I was worried about weight, both for travel as well as moving underwater. I was pretty pleased with the eventual weight — about ten pounds for the frame and another five for the cameras. Underwater, it was slightly negatively buoyant, but not crazy — I could snorkle with the camera with just the buoyancy of my wetsuit and have no trouble staying above the water. With full scuba gear, it was easy to compensate for the added weight of the camera with our BCDs.
I did several tests in Providence before we left on the trip. The first was a simple video test in my lab, to work out the process of recording data from all cameras, getting the footage off, and processing into a lightfield format. It turned out pretty well.
We're getting somewhere.
The Light field Camera Build:
Carlos: After some back and forth, discussing possible underwater conditions and Alex's technical concerns, this came to life:
This is a 12 ft, 32 camera slingin', hologram producing, monster. In 6 months we'll look back and wonder how we:
- Got Down to Ecuador with this
- Actually made it the dive site with this in tow
- Made it back out of the country, intact.
What does your expedition title mean and why are you trying so hard to be clever?
We're (team background below) attempting to film Giant Manta Rays at their largest known congregation site in the world, Isla De La Plata, Ecuador and eventually display them in Holographic form. There's so many reasons to do this, but most importantly, why not?
The second part, we're just like this, trying to work through it.
How are we going to do this or at least try to do this?
Carlos/Steer Digital: With our custom designed + built 32 camera rig, powered by GoPro (can we get some t-shirts or a hat, GoPro...), then composite that footage into a file that the Hologram display (lookingglassfactory.com) can then project into a 3D Hologram.
Alex/Looking Glass: Carlos and I met about three months ago. I was just starting to build some experimental holographic cameras, and someone told me that underwater filmmakers were diving with unusual cameras, so I asked around for someone to talk to and got introduced to Carlos. We spent a nice afternoon playing with some of our holographic demos and talking about capturing holographic scenes underwater, and Carlos off-handedly mentioned that he was planning a trip to Ecuador to film Giant Manta Rays in a few months, and if I could make a camera, they could take it down and shoot.
I took about a millisecond before blurting “I’ll make you a camera!” It seemed like a no-brainer — the right mix of science and adventure. There was only one problem — I had never made a holographic video camera before, and the trip was coming right up. The next two months slammed on the gas and accelerated into a whirlwind of activity as I raced to research, invent and test an underwater holographic video camera.
Our displays work by showing many simultaneous views of a 3D scene. Before this project, we had only worked with virtual 3D scenes and virtual cameras, and all this is very abstract and mathematical until it dawned on me that, to capture a manta ray-sized subject, I would have to build a manta ray-sized camera, with thirty-two cameras bolted onto it, and find a way to swim around underwater with this monstrosity. I needed a way to think about everything that is involved in translating our virtual cameras into real-world cameras, so I built a little manta-ray simulator.
This simulator let me quickly try out different camera configurations, take virtual light-field photos of virtual manta rays, and see how they look in the device. I played with the lens field of view, the distance from the subject, the total size of the volume I wanted to film, and the spacing between cameras. In the end, I wanted to figure out how large the camera needed to be: could we film a manta ray without a manta ray-sized camera?
To make things worse, I had scuba dived before, but I had never held anything in my hands, so I didn’t have a good reference for what was reasonable and what was crazy. Carlos and I bounced around a lot of ideas on this, and I started bugging the divemasters at my local diveshop to help me up the steep learning curve of building underwater machinery. I also didn’t really know anything about manta behavior underwater — do they stay in one place? How fast do they move? Do we need to follow them? I’d call up Carlos, pepper him with questions, and then sit down and try to draw up an improved design for the lightfield camera.
In the end, I settled on two different camera designs: an eight-foot-long camera and a twelve-foot-long camera. The more I worked with simulations, the more I appreciated that I could trade off camera resolution to adjust perspective by cropping down each of my camera shots to simulate a narrow field-of-view. It’s a cheap trick, but it gave me a lot of power to adjust the focus plane and size of my shot after I shot it — the ultimate in “uhhh, we’ll fix that in post.” Lightfield Camera v1 in Looking Glass’ (messy) skunkworks lab
Once I’d settled on a design, the only thing left to do was order thirty-two GoPros and dive housings and a lot of aluminum extrusion and figure out how to hold them together onto something we could actually use in the water and figure out how on earth to synchronize and edit thirty-two simultaneous video files and get the whole thing to show up in our holographic displays. Also, I should mention, the last time I dove was when G.W. Bush was president. There was a lot to do before the trip.
Carlos: Yes, all that above. When Alex proposed this, I had to go check out his studio first and see what I was getting myself into. I visited Isla De La Plata in 2017 and had the opportunity to dive with marine biologist Michel Guerrero for about a week and knew instantly I'd be back. Gliding under large Manta Rays (Mobula Birostris) is an incredible experience and much different then my experience with large predator diving (Tigers, Giant Hammerheads, Bulls, ). For those species, it seemed like almost as soon as the bait/food source was gone, they would soon be gone as well. With Giant Mantas they spend what seems like in an inordinate amount of time deciding what you are and what to make of you, and seemingly without a food source motivation, in other words, they're checking you out.
This intense (they can spend upwards of 30 minutes gliding along with you) bi-directional curiosity makes this experience surreal and worth capturing.
Having said all that, you don't want to introduce something into that underwater environment, where the currents can pick up, that's going to add tons of drag to your movements. Other than pure luck, you'll have very little chance of pulling this off without some decent design thinking going into the build of this rig.
Why are we doing this?
We ask ourselves this every. single. day.
But really, we want to experiment. This is complicated and messy, with no guarantees to the final outcome, but there's a good chance that this expedition will result in really engaging content.
If we can figure this out, this could lead to new forms of education, learning and understanding.
It's just one step closer to bringing the world to you, or maybe that kid in the Bronx, Bogotá, or Milwaukee, who just needs to see the lyrical flow of a Giant Mantas' wings up close to decide she's going to spend some extra time finding out what that being really is and maybe deciding it's worth protecting.
We believe this is the first time that any team has dived with a portable lightfield capture device (Guinness, hit us up. Either one.)
And that means this is
probably definitely going to be marked with daily failures, we hope small and surmountable, but we will learn a lot; that's guaranteed.
We'll journal our discoveries whether big, small, technical or animal. Who knows what we will encounter at Isla De La Plata.
Our expedition begins on August 23rd and the clock is ticking to complete the build, test it here in Rhode Island and practice through some dual dive scenarios.
Alex Hornstein: Inventor/Co-Founder/CTO at Looking Glass Factory
Carlos Toro: Director / Producer / Founder at steerdigitalmedia.com IMDB profile
Michel Guerrero: Marine Biologist / Founder at Proyecto Mantas Ecuador
Contribute to this expedition
Thank You for Your Contribution!