Gurukul My Project Guide

Introduction

The web browser is the ubiquitous platform for this web communications project. Unfortunately, current browsers lack the full capability to support multimedia communication, hence plugins and extensions are needed. We need to enumerate and evaluate the available platform options.

The critical requirements of the platform are: ability to use audio and video devices and ability to use UDP transport along with HTTP (TCP/TLS). A platform that facilitates portability across different OS and hardware, as well as easily allows adding new components (e.g., a new audio codec) is desirable. The primary protocol on the web, HTTP, is fundamentally client-server, hence the system comprises of clients as well as servers. However, this document deals with only client platform issues and options.

The potential options are listed below:

1. Existing Browser Plugin: Use an existing plugin such as Flash Player.
2. Browser Modification: Modify an existing open source browser to incorporate the missing features needed for multimedia communications.
3. Separate Application: Use a separate application or service running on user's computer that allows web browser to do multimedia communications.
4. New Browser Plugin: Develop a new plugin that works independently or in conjunction with existing plugin.

Existing Browser Plugin

For web video communication, Flash Player comes to mind immediately. The platform provides existing components and widgets, and a new communication component can fit naturally with the existing ones. The numerous Flex programmers can contribute to building communication applications making it a success.

+ Portable on all major OS and browsers.
+ Already available on almost all users' PCs. No separate installation needed.
+ Huge number of Flex developers can contribute. Web developers are already familiar with building Flex application with video. There are number of available tools and code snippets.
+ Independent of browser war. Browser incompatibility is not much an issue.
-- Limited connection API: no UDP socket available to Flex application programmers. Also, there is no incoming TCP socket connection.
-- Limited media access: no access to captured audio and video to Flex application programmers. Media transport happens directly between Flash Player instances and backend server without going through the application. Although the audio codec (e.g., Speex) is open, the video codec (e.g., Sorenson) used by camera in Flash Player is patented.
-- Not available yet on iPhone. Challenging on other smart phones.

It seems using a Flash Player is not enough for this project. Other alternatives to Flash Player such as Silverlight and SVG/HTML5 are not yet mature to be used now and potentially suffer from the same problems of limited connection and media access. Another alternative is the Adobe Integrated Runtime (AIR) which is described later.

Browser Modification

In this approach, modify an existing browser tool such as Mozilla Firefox to incorporate the missing features. These features can be exposed to applications (Javascript and HTML) using new objects similar to how Ajax uses XMLHttpRequest object. Depending on the granularity of the API, it could be low level Camera, Microphone, MediaSession objects or high level MultimediaCall object. Perhaps new tags can be added in a future version of HTML to facilitate a communication component, similar to how video tag will facilitate video playback in HTML 5.

+ No other external dependency other than a web browser.
+ Potential to contribute to the standardization of future HTML.
+ Huge number of JavaScript developers can contribute to huge success. Web developers are already familiar with JavaScript for programming and HTML/CSS for view layout.
-- Need to get people to use the modified browser. It may take forever to be ubiquitous.
-- Development cost: Modifying a browser implementation is challenging.
-- Portability: dealing with portable audio and video devices interface is challenging.
-- Multiple instances of the browser need to share the same resources (devices).

The major challenge in this approach is that people are reluctant to change their web browser.

Separate Application

Instead of modifying the browser, one can build a separate standalone application or service that runs on user's PC, and allows the web pages to use the features. In this approach, the web page can embed a Flash application which will communicate with the standalone application or service outside the browser on the local machine. Although the user interface is in web browser, the actual processing happens in the separate application. This approach is similar to the Gmail plugin that enables video chat from within the browser.

+ Flexibility: a separate application has a flexibility of using any transport protocol or adding any new audio/video codecs.
+ Portability: the proof-of-concept application can be built in a portable programming language such as Python or Java.
+ Portability: if Flash Player is used for front end web application, then it can deal with device access, and communicate the captured audio and video packets to the standalone application. Similarly, the received packets can be sent to the front end application for display and playback.
-- Need to get people to install the application.
-- Multiple users need to access the application on a multi-user system. The challenge is to provide an OS equivalent security and access control.

This approach has the benefits of the separate application as well as existing plugin approaches: portability, flexibility, existing developers' community. The main challenge is to have people install the separate application.

As mentioned before, AIR can be used to implement the separate desktop application. Although AIR provides access to UDP socket to the Flex application programmers and raw data captured from the microphone, there are other limitations. AIR does not give access to encoded audio data or video captured from camera. It may be possible to implement voice-only calls using primitive codecs, it becomes difficult to do Speex-based audio call or video call without support from an external application.

New Browser Plugin

In this approach, instead of modifying the browser, a new browser plugin is developed which fills in the feature gap -- UDP socket and audio/video codecs -- and allows other web applications to use these features. Instead of running a separate application, the separate plugin inside the browser handles all the connection and media access issues. Thus, a Flash application can connect to this plugin to implement the UDP socket or incorporate audio/video codecs.

-- Are there any limitations of plugins?

The main challenge is portability. Depending on what the plugin is allowed to do, we are also limited in flexibility.

Conclusion

I think the separate application approach makes the most sense in the short term. The application can provide API for control and media access over HTTP (for control) as well as RTMP (for media) to web applications. The web application developers can use this to build contact list, communicator user interface, or video conference rooms. The front end application can be in JavaScript or ActionScript (Flash Player), and accessible from within a web page as well as browser extensions, e.g., Firefox sidebar for contact list. The project can provide sample client applications that shows how to use the API provided by the separate application.