17 Apr SECOND SCREEN – Companion Second Screen Application
1 – INTRODUCTION
Television is still one of the main source of home entertainment. However, the way we watch TV has changed significantly in recent years, starting with the different recording tools, which allow users to watch favorite tv programs at any time, regardless of the television broadcaster, up to simultaneous interaction with mobile devices: laptops, smartphones and tablets are commonly used while watching TV.
The viewers, in addition to chatting or checking emails, can look for information about the actors, locations or previous episodes of the tv shows, etc.
The “Second Screen” is an important investment area for companies: finding new ways to exploit this new technology, which offers new possibilities for marketing and market analysis, as well as raise viewer’s interest in the tv shows themselves.
On the other hand, the interaction between mobile applications and tv programs presents complex technological aspects that make it an area that is still not so much explored.
From these considerations , the idea of this work, which aims to be the experimentation of an application platform capable to offer supplementary contents to viewers, through an ordinary mobile device (smartphone/tablet), while watching a television program or while listening to a radio show.
The Second Screen system provides a television broadcaster with the possibility to offer its viewers, not only the transmission of ordinary television programs, but also a specific Smartphone App to access additional information contents.
These contents come from a database that the television broadcaster keeps updated, containing associated information with specific scenes or moments of the tv shows (live tv shows are not included).
Once started, the App activates the microphone in order to “listen” to the surrounding environment. When the audio sequence of a scene is recorded and the data are processed, it displays its associated contents.
The Second Screen system requires the creation and maintenance of a database of multimedia contents, linked with specific portions of the tv programs.
It is necessary to provide an archive to preserve:
- Tv programs list
- For each tv program, the portions (scenes) to which associate a content
- For each scene, the associated content, which can be simple text or even a formatted text, containing images and video sequences.
It is necessary to provide a back-office application that allows you to manage the contained information and keep it updated:
- Manage tv programs
- Define the scenes of interest for each tv program: select the start and duration of each scene within the program.
- Edit the contents associated with the scenes.
The front-end for end-users will be an elementary use App, intended for a large audience and not necessarily technically advanced.
Once activated, the App must use the device’s microphone (phone or tablet) to listen to the surrounding audio and, if it recognizes a sequence recorded in the database, it will display it on the screen.
2 – Second Screen Solution
The system consists of three main components:
1. The database of contents associated with audio tracks
2. A web application intended for the operators of the television station, for the management of contents and for their distribution to the App
3. An App for the use of the Second Screen by the viewer, wich has access to the contents of the database and is able to recognize audio sequences using the microphone of the mobile device where it runs.
2.1 – Fingerprinting algorithm for audio tracks
There are two main methodologies for solving the problem of recognizing audio sequences:
1. Watermarking
2. Fingerprinting
Watermarking consists of superimposing on an audio track, in correspondence of the scenes we want to identify, short sound sequences built from not human perceivable frequencies, but recognizable via software.
Fingerprinting does not make any changes to the audio tracks, but extracts few essential information from the selected scenes to be able and subsequently identify them, even in the presence of noise or distortion.
Both watermarking and fingerprinting often use the same techniques to reduce the size of the recognition problem, which typically include the Fourier transform (FFT) to filter the high frequencies (normally containing noise), the Wavelet to represent the signal in compact form, as well as various numerical hashing methods to minimize the amount of information involved.
Although watermarking usually shows better performance than fingerprinting – as different watermarks can be built specifically to make them easily distinguishable from each other, while remaining imperceptible to the human ear – in the context of Second Screen application, a method based on fingerprint is more easily applicable, since it does not require you to modify the audio tracks of contents already recorded and in possession of the television broadcaster.
For this reason our Second Screen system uses a fingerprinting algorithm.
2.2 – Data model
The following diagram describes the data model of the system support database.
It is possible to define multiple scenes for each tv program and, for each of them, associate an integrative and interactive content, which in the demo application is a simple text, but can easily consist of HTML text enriched with images or other multimedia elements. Each scene must have a duration of a few seconds (around 10s-20s), for which the fingerprinting algorithm will extract a certain number of “fingerprints“, which will be stored in the database.
2.3 – Management portal
The portal provides the back-office functions, necessary to manage the contents of the database.
Realized in Microsoft .NET 4.7 programming language, the portal is intended to be installed in server environments with Microsoft Windows Server 2012 or higher operating system version.
2.3.1 – Home Page
From the home page, clicking on the “Movies” menu you can enter to the page for managing audio/video titles and related extra contents, which can be displayed on the Second Screen.
2.3.2 – Video tracks management
The window presents the essential management functions: submit/edit/delete of an audio/video content and upload of the relative file, which must be provided in .mp4 format.
By clicking on the link on the right of each title, you can access the page for managing the extra contents associated with the film or television program.
2.3.3 – Management of scenes with extra contents
This is the page that allows you to define the textual contents displayed on the Second Screen.
Start and Duration are the properties, expressed in seconds, which allow you to define the portion of a scene on which the fingerprints will be calculated. Average duration of these segments must be between 10” and 20” in order to obtain sufficient reliability during recognition by the Second Screen App.
The audio sequence used for fingerprints must partially precede or superimpose the beginning of the scene to which the extra content refers, since the App will need to “listen” to a good part of this audio sequence, before recognizing it and viewing its associated contents.
For example, if you want to associate extra contents with a scene that starts at 45th”, you can choose a 10″ “duration” that starts at 40th”. After a few seconds from the beginning of the scene, the viewer will see the supplementary information appearing on his Second Screen.
2.4 – Second Screen’s App
The HN Second Screen App intended for end-users (the tv viewers) has been developed using the Microsoft Xamarin framework, a solution that allows you to easily develop mobile applications for both Android and IOS.
Being still a feasibility study phase, the App presents the necessary features to demonstrate the effectiveness of the fingerprinting algorithm.
2.4.1 – Program selection
The user selects the television program from the main menu, which reports the titles surveyed in the database by the operators of the broadcaster, using the management portal.
2.4.2 – Second Screen
Once a title has been selected, the Second Screen operation is activated by clicking on the “start listening” button.
The button allows the viewer to alternately pause the audio recognition and reactivate it.
When the Second Screen is activated, the App “listens”, using the microphone built into the tablet or smartphone and, while the viewer watches the program on any other device (television, computer, etc.) it displays the associated textual contents to the scenes, which are gradually identified by the App.
