The New Replay
Machine vision is helping viewers craft their own perspective on the big game.
Replay Technologies is developing a new level of interactivity for broadcast video programming. This new approach to its machine-vision-driven freeD system has the potential to allow you, the viewer, to direct the scene you’re watching.
The company has pioneered 3D video of sporting events, allowing show producers to move virtual cameras through three-dimensional representations of live events within seconds of play. The result? The re-creation of incredible touchdown passes and awe-inspiring slam-dunk replays.
Today, millions of people across the United States have already seen this new reality through Sunday Night NFL Football games, Yankees Stadium, the 2014 NBA All-Stars Weekend, and ATP Masters Tennis. Replay wants to deliver that power to the viewers.
“Initially, we put this technology in the hands of directors and producers—the ones who know exactly how to build a story with this technology. But we see the next revolution coming to the consumer side,” says Oren H. Yogev, co-founder and CEO of Replay Technologies Inc. “Everyone has a camera on their smartphone, and so everyone is an amateur photographer. We learn and share with our family and friends. With this new technology, we become amateur directors. In this market you want everything to be viral. You want as many users and as much exposure as possible.”
Pushing the Limits with Machine Vision
The freeD system that makes this all possible uses high performance area cameras, heavy duty vision processing boards, and advanced algorithms to render photo-realistic, real-time scenes. It’s often referred to as “bullet time” footage, named after the famous 360-degree shot of Keanu Reeves dodging bullets in The Matrix. In 1999, this shot was achieved using a bulky circular rig in a carefully lit green screen studio.
The freeD system takes on a much more complicated task, shooting in a stadium where there is much less control of lighting conditions and environment. The footage is taken by multiple cameras positioned around the most pivotal areas of the field (the goal posts/hoops/tee-offs), piped via fiber to multiple acquisition boards in a server cluster, and converted into a 3D point cloud database of image information which can then be rendered out by Replay’s sophisticated software at whatever viewing angle the director desires …all in 30-45 seconds.
Letting the Audience Run the Show
The result will be a new level of interactive video programming. “We are already working with a national network for a huge sporting event next year. As you watch, you’ll have the opportunity to decide whether you want the shot chosen by the show’s director, or a shot of your own choice. You can be up in the stadium or right on the field with the players.”
The current plan is to offer this new content as a second screen experience. With the original show remaining on your TV screen, the program will indicate when there is freeD content available—a touchdown or reaction shot. With an app or player, the viewer can interact and create a different angle and shot sequence all from this second screen. As the ‘director’ of this new shot, you’ll be able to share it as well. “We already have a prototype of a player, and once we have the 3D data, we can let people play with it. By the end of this year, we may be able to launch the first player.”
Replay Technologies is working at the ragged edges of what can be done in terms of cameras, graphics processing, and bandwidth. Yogev wants the 3D shots to be full motion video, and in real time. Though it hasn’t been done, yet, the possibility remains. “But we’re not sitting on the fence, waiting for IBM or Intel to give us a server for two billion dollars. We aren’t NASA. We’re moving fast on our own. We can already do what they were doing in The Matrix movie ten years ago, but in real life, in almost real-time. As soon as the technology can get there, we are ready.”
Yogev says he and his team knew that machine vision was the only solution from the beginning. “You need to make it accessible to people—and for yourself. We came from other industries that used machine vision and knew what was possible. We made it work for us. For the money you pay, machine vision cameras are better than what people are using elsewhere. Other cameras are too expensive and complicated.”
While Replay’s systems require the use of 10 to 25 high-resolution machine vision cameras in a stadium, Yogev says that “they aren’t that expensive when considered against other television production costs. They make a lot of sense in terms of what you can accomplish and the economics behind them.”
Mobile – The Next Frontier
While Replay is committed to imaging with machine vision cameras, it won’t stop there: “We can’t yet use data from individual camera phones, because we don’t know the exact location of each camera. But once we have location information, we can explore this ability to let people play in space and time to direct their own shows.”
“Once they understand the power they have in their hands, it will open up entirely new opportunities.”
Creative Vision Engineers 