Introduction
Sporting events today are rife with advertising, from commercials on screens to static company logos on stadium billboards. The airwaves span multiple countries with different brands and advertising laws. Thanks to breakthroughs in AI and AR technology, it is now possible to customize the information on display for each audience, directly during the live broadcast of a match.
Evolution of Sports Advertising
Advertising in sports arenas was initially static and intended for the people attending a particular event in a particular city. Later, LED screens appeared, showing alternating adverts. The next evolution was inserting a green screen into the billboard spots for a fraction of a second, allowing AI to define the space and insert unique text.
Nowadays, you can replace anything with anything. The challenge is to ensure that the replacement is done discreetly and realistically. Major players in this market use special cameras and specialized sensors to discern positioning and alignment information. Popular solutions still rely on these hardwired components.
Nowadays, you can replace anything with anything. The challenge is to ensure that the replacement is done discreetly and realistically. Major players in this market use special cameras and specialized sensors to discern positioning and alignment information. Popular solutions still rely on these hardwired components.
Challenges of Current Technology
All this requires enormous computing power, as the inputs and the original broadcast are processed by specialized software online. Servers for computing are connected to the cameras and sensors, making this equipment expensive and the market closed. This is where AI and cloud GPU servers come to the rescue.
Introducing PTF Lab
PTF Lab has developed its own technology for implementing virtual advertising and integrating digital content (like augmented reality) in a multi-regional mode. Our solution promises seamless integration of adverts directly into the video stream.
Our goals are noble, understandable, and quite achievable:
Moreover, we aim to surpass the solutions offered by monopoly giants in terms of flexibility, allowing for "virtual adverts" during replays and from any camera angle.
Our goals are noble, understandable, and quite achievable:
- To move away from expensive proprietary equipment and complex setups, shifting the task of advertising placement and frame construction to AI, which takes into account overlapping people and objects in the frame.
- To cover relatively small events (such as arena fights) and bring the technology to the masses.
- Ultimately, to make sports advertising accessible and relatively inexpensive.
Moreover, we aim to surpass the solutions offered by monopoly giants in terms of flexibility, allowing for "virtual adverts" during replays and from any camera angle.
How Does It Work?
The video signal from a sporting event venue can be processed using computing power not only at the venue itself but also in the cloud. This allows for flexible load distribution and the choice of when to apply adverts: before or during broadcasting, taking into account different markets. Working with cloud services also allows the use of advertising in locations where it is impossible to bring a server.
Object segmentation is based on the neural network architecture from U-Net. Neural networks are responsible for the location of objects and for detecting and comparing key points. However, the task is non-trivial, so all the solutions and neural networks had to be reworked and trained for use.
Object segmentation is based on the neural network architecture from U-Net. Neural networks are responsible for the location of objects and for detecting and comparing key points. However, the task is non-trivial, so all the solutions and neural networks had to be reworked and trained for use.
Challenges in Martial Arts Broadcasting
In martial arts broadcasts, where everything is unpredictable, the task is especially difficult. Light sources, shadows, camera angles, the grid overlapping sponsor logos, and the bodies of the fighters and referees all pose challenges.
Neural networks are not used everywhere. Sometimes, to solve a problem, it is enough to show ingenuity and use simple algorithms. For example, tracking algorithms combine neural network methods and systems of linear and nonlinear equations.
A significant part of the GPU is taken up by segmentation. The better the detection of people and objects in the frame and their separation by plans and type, the more natural and attractive the frame will look after the advertising overlay.
A separate task is related to lighting and shadows, which must be taken into account in augmented reality when rendering a scene. The realism of shadows is a key element in assessing the "believability" of the picture.
Neural networks are not used everywhere. Sometimes, to solve a problem, it is enough to show ingenuity and use simple algorithms. For example, tracking algorithms combine neural network methods and systems of linear and nonlinear equations.
A significant part of the GPU is taken up by segmentation. The better the detection of people and objects in the frame and their separation by plans and type, the more natural and attractive the frame will look after the advertising overlay.
A separate task is related to lighting and shadows, which must be taken into account in augmented reality when rendering a scene. The realism of shadows is a key element in assessing the "believability" of the picture.
Training Neural Networks
Traditionally, sports neural networks are trained on real broadcasts using human markings and synthetic models. Blender helps by building 3D models of the ring, fighters, referees, and getting both rendered real footage from the right angles and the segmentation mask, or the position of objects and cameras needed for training. Markup of real data is time-consuming and expensive but of high quality for specific venues or types of competitions. Synthetic data with less realism provides more data for training.
Adapting to Different Venues
The main difficulty is that the venues can vary. In one case, it will be a boxing ring with ropes; in another case, it will be an arena with mesh walls, each creating segmentation difficulties.
Camera tracking and advert position are determined by comparing the point cloud from the 3D model of the venue with their actual position in the frame. This allows for determining the position even for manual cameras with chaotic movement. After reconstructing the 3D frame from 2D, the direct rendering of the advertisement in the 3D engine is performed and combined with the video broadcast frame.
A 3D scene has to be built before starting work, creating a virtual copy of the venue in the frame, into which real people and objects are fit through render masks. It sounds complicated, but with the right power and optimized neural networks, it is possible to perform these tasks instantly and seamlessly.
Camera tracking and advert position are determined by comparing the point cloud from the 3D model of the venue with their actual position in the frame. This allows for determining the position even for manual cameras with chaotic movement. After reconstructing the 3D frame from 2D, the direct rendering of the advertisement in the 3D engine is performed and combined with the video broadcast frame.
A 3D scene has to be built before starting work, creating a virtual copy of the venue in the frame, into which real people and objects are fit through render masks. It sounds complicated, but with the right power and optimized neural networks, it is possible to perform these tasks instantly and seamlessly.
Scaling with Cloud Resources
PTF Lab has its own servers but prefers using remote resources, as the service provider's engineers are responsible for equipment availability. This allows the company to allocate fewer resources to this and focus on their core activities. Cost-effective options are always preferable.
Also, the capacity required by the company is constantly growing. If necessary, it can be easily scaled up by renting more (up to and including changing the server configuration to suit the company's needs).
In the future, the startup may need a lot of cloud capacity. It is easier to rent them than to buy and sell physical servers when the demand for their services rises and falls.
Also, the capacity required by the company is constantly growing. If necessary, it can be easily scaled up by renting more (up to and including changing the server configuration to suit the company's needs).
In the future, the startup may need a lot of cloud capacity. It is easier to rent them than to buy and sell physical servers when the demand for their services rises and falls.
Conclusion
PTF Lab is at the beginning of its journey, but our results are already promising. We are committed to overcoming the monopoly of sports augmented reality and making high-tech sports technology accessible for all. The witty David always defeats the clumsy proprietary Goliath.