Developing Interactive TV Applications for tvOS

Children’s learning shows try to engage kids by showing them content and then prompting them to respond back before continuing with the rest of the content. Dora the Explorer is a great example. Dora will ask “How many apples can I fit in my bag?” and the video will pause to allow kids at home to answer. Then the video will continue on the same track regardless of whether the answer given by the kids was correct.

This is already how kids interact with the content and how publishers release the content–the pause effect right now is just arbitrary. This is where we make it interactive. The intent is to provide an interactive TV experience for kids where the content only moves forward if the kids interact with it using their voice, and different clips are played based on responses.

As we all know, Apple is already a giant in the electronic mobile gadgets arena. They have recently come up with the idea of creating applications for their Apple TV service. For developers to be able to play around with Apple TV apps, Apple has provided tvOS, which is a huge thrill.

Parents may want to provide their kids with an interactive way to learn, such as through videos will call and response features. Apple has provided various tools (TV Simulators) and instruments (Performance Optimization) to create these kinds of applications using tvOS.

How does tvOS work?

According to Apple: “We’ve reimagined the TV experience with tvOS — an innovative TV platform that redefines what can be done in the living room. The new Apple TV features a built-in App Store making it more entertaining than ever. Now you can deliver incredible and immersive apps and games on the big screen using the tvOS SDK.”

Now we can begin to create something interactive for kids for tvOS SDK.

Now the big question: how does it work?

Getting Started: A Brief Overview

As the diagram above illustrates, we create a web service that takes voice as an input, converts it into text, and sends it as a response.

The tvOS application looks up to the web service for a response and plays different clips based on the response:

• Correct answer: notifies that the answer is correct and moves onto the next question
• Wrong answer: skips directly to the next question
• No answer received: keeps repeating the same question until a response is received

Time to play with some code

Tools Used: Xcode 7.1 beta with tvOS SDK.

Steps:

1. In Xcode, we create a Single View application.
2. Add AVFoundation framework into the application.
3. Import the files into the View Controller.

There are a few mandatory methods for playing a video on tvOS:

The tvOS application looks up to the web service for a response and plays different clips based on the response:

AVPlayer Object: initialize it using the video URL

-(void)addObserver: In this method, use addObserver on AVPlayer on specific intervals that correspond to the questions asked in the video. We are calling the service using the URL at those specific times.

-(void)connectionMade:(Completionhandler)handler: This method returns the string that we say into the browser. We can check that the string has been returned through this method whether it is correct or not. If correct, the video plays on and keeps asking new questions. If wrong, the video skips the part in which it says the correct answer and asks a new question. No response, and the video will be played in a loop in which only background music is playing and we check the connection. The method will continue in a recursive manner until we get an answer.

Below is a demo video showing the functions of the interactive app.

The Challenge with tvOS

tvOS was recently launched by Apple. Though we can create applications for Apple TV, as far as we have found, we can’t provide voice input to our tvOS application. Yes, Apple provides a dual mic in their TV remote, so we can get the voice input by using these mics. However, the problem is that Apple only provided the hardware, the mic, not the API to get the callback to listen the voice input into our application. They have provided the voice input for Siri only.

However, we have come up with a work-around to overcome this challenge. We can make an iOS application that can be used as a communication medium between the user of our application and AppleTV. For example, kids can speak into their iphone, iPad, or iPod and the iOS application will convert the voice into text and send it to the tvOS app.

Can we implement this idea for Android TV as well?

Android TV apps follow the same structure as those for phones and tablets except for the application theme. This means that we can modify our existing apps to run on TV devices or create new apps based on what you already know about building apps for the Android platform.

The following are the main components we must use to create an app that runs on TV hardware/devices:

• TV Activity (REQUIRED): In your application manifest, declare an activity that is intended to run on TV devices.
• TV Support Libraries (OPTIONAL): There are several support libs available for TV devices tha tprovide widgets for building UI.

Before we begin building apps for TV, we must have the following tools and API in our development environment:

• Android SDK tools: The updated SDK tools (version 24.0.0 or higher) enable you to build and test apps for TV.
• Android SDK: Update your SDK with Android 5.0 (API 21) or higher which provides new APIs for TV apps.
• Create or update your app project: In order to access new APIs for TV devices, you must create a project or modify an existing project that targets Android 5.0 (API level 21) or higher.

There are four major classes in project modeling:

1. Video: A video class encapsulates a list of story objects
2. Story: A story class contains the story start, stop, loop (repeat interval), time (in milliseconds), and story’s answer.
3. PlayBack: This class contains a video and a pointer for currently playing a story.
4. PlayList: A PlayList class contains a list of videos to be played, as well as behaviors like loadNext(), loadPrevious(), and getCurrent(), which returns PlayBack objects.

Code Snippet

We created a VideoPlayerActivity class to control the video playback as per our need. So when a user clicks on the app launcher icon, this activity gets started and plays video.

Web service implementation code

Like a tvOS app, our Android TV app also connects with the server for voice input. For server communication we are using the OKHTTP library. The code for communication with the server is as follows:

There are two classes: HTTPBackgroundTask and HTTPUtil, which completely handles communication operation.

This HTTPBackgroundTask class is responsible for continuously looking up the server response. As soon as it gets a server response, it gives a call back to the video controller to play the video accordingly.

How to run the Android app

On an actual device (Android TV):

After app installation, you will see the app launcher icon on the app launcher area. Click on the launcher icon and your app will run immediately.

There are three ways to install an app on Google TV:

1. Through Google Play – Open Google Play on your computer and select the app. Click on the Install button and choose device “Your Android TV.”
2. Through other apps – Using an ES file explorer, load your APK on Cloud, then open the ES file app and choose Cloud. Download the app and install through remote. Note: You would need to enable security to allow you to install from unknown sources. On the SideLauncher app, it will show any third party apps installed by you in a group.
3. Through micro-USB cable: Load APK into USB flash drive and connect it with Google TV. Open ES file explorer and install the app from the flash drive.

On a virtual device (simulator):

The AVD (Android Virtual Device) Manager in the Android SDK provides the device definitions that allow you to create virtual TV devices for running and testing your applications.

To create a virtual TV device:

1. Start the AVD Manager. For more information, see the AVD Manager help.
2. In the AVD Manager dialogue, click the Device Definitions tab.
3. Select one of the Android TV device definitions and click Create AVD.
4. Select the emulator options and click OK to create the AVD.
5. Compile your TV application in your development environment.
6. Run the application from your development environment and choose the TV virtual device as the target.

Have questions about building tvOS apps or apps for Android TV? Let me know in the comments section below.