Development of a System of Five Neural Networks for TV Broadcast Analysis

For video content analysis, we integrated five neural networks into the project:

• The first YOLOv8 detects logos.
• The second YOLOv8 identifies common objects (umbrella, ball, person, sneakers, dog, etc.).
• Rev AI transcribes speech and sends the text to ChatGPT.
• ChatGPT extracts mentions of brands, cities, and celebrities from the transcript and determines sentiment (positive/negative attitude of the speaker toward an entity).
• Tesseract recognizes static text within the video frame, such as subtitles, captions, and text-based logos.

TV broadcast analysis needs to be fast, making performance a major challenge. To ensure smooth operation, we implemented a microservice architecture.

The product consists of multiple services, each handling a specific task. This approach makes the project highly scalable, allowing the owner to add new services to enhance performance and increase network capacity as needed.

To train YOLOv8 to recognize standard objects like people, cars, animals, and clothing, we used the COCO dataset without making any modifications.

Logo detection was more challenging, but we ultimately chose the open-source OpenLogo dataset, which contains data on 352 logos and 27,000 labeled images.

After discussions with the client, we decided to add 40 more logos to our dataset. To train the neural network to detect each logo, we gathered at least 50 images per logo and augmented them threefold.

We also had to adjust the original OpenLogo dataset. The number of images per logo was uneven—some logos had only 20 images, while others had 150.

To fix this, we balanced the dataset. Before training, we identified logos with fewer images than others. During training, we penalized the model more heavily when it missed logos with fewer labeled images.

Currently, our service analyzes video files uploaded by users. It takes 5 minutes to process a 10-minute video. In the near future, we plan to integrate WebSocket support and expand the gateway to enable live TV broadcast analysis.

Next, we will integrate the project's backend as an API into the client’s digital products. This will allow European marketers to use our service to launch ads at the most relevant moments.

For example, if the main character in an action movie is wearing Nike sneakers, why not reinforce the impact with a Nike commercial right after the film?