Survey on Multimedia Technologies for Mobile Learning Applications

Abstract

Mobile technologies are developing very fast. This paper presents a survey on multimedia technologies for mobile learning applications, focusing on multimedia programming techniques for Windows Mobile, Symbian, and Java ME. [PUBLICATION ABSTRACT]

Headnote

Mobile technologies are developing very fast. This paper presents a survey on multimedia technologies for mobile learning applications, focusing on multimedia programming techniques for Windows Mobile, Symbian, and Java ME.

Keywords: multimedia, mobile devices, mobile applications, mobile learning

(ProQuest: ... denotes formulae omitted.)

1 Introduction

Mobile learning is implemented using different technologies and platforms. Each implementation has specific characteristics in terms of user interface and content and influences the quality management process. The software required for mobile learning process is a Web browser or a specific application, that can be standalone or a client application.

Web-based platforms are very common in mobile learning process. Figure 1 depicts the architecture of this platform.

The advantages of using Web-based mobile learning platform are:

* User's minimal effort to set up the application;

* The user doesn't need to learn how to use a new application; the client is a simple mobile Web browser (like Pocket Internet Explorer of Opera Mobile);

* E-learning content can be easily updated on the server.

There are also some disadvantages:

* Not so rich user interface (browser-based);

* Multimedia content is limited;

* Possible additional costs for traffic usage.

Distributed platforms have a similar architecture as Web-based platforms, but the client application is a rich application and not a simple mobile Web browser. The advantages of this platform are:

* Rich user interface;

* Support for multimedia content;

* E-learning content can be easily updated on the server;

There are also some disadvantages:

* The user need to install and setup the client application;

* The user have to learn how to use the application;

* Possible additional costs for traffic usage.

Standalone applications do not require network access. They are similar to the client applications for distributed platforms. The main disadvantage is that the update of content needs access to device.

Errors and applications failures influence the learning process. Distributed applications and Web-based requires more test because of network access. The test plans need to include use cases on various hardware platforms, operating systems, networks bandwidths in order to cover a large number of scenarios.

Almost all mobile operating systems have support for multimedia applications development. Some technologies and libraries exist in desktop versions (like Open GL, DirectX etc.), other are specific to some operating system [2], [3], [4], [5].

Multimedia m-applications (figure 2) access audio, graphics and media API through operating system or third party libraries. These API communicate with multimedia related hardware through device drivers.

The display resolution, audio and video system, 3D support, and operating system influence the multimedia application development for mobile devices.

The audio and video resources involve playback and recording capabilities.

2 Java Micro Edition (Java ME)

Java Micro Edition platform is intended for all mobile devices that have limited hardware and software resources. In order to allow the development of application for these devices, this platform has limited framework support comparing to its big brother, Java Standard Development Kit. Despite the early characteristics of devices, present and future mobile technologies allow software developers to design and implement rich multimedia content. In order to support this kind of resources, Java ME platform includes a package, the Mobile Media API (MMAPI), [6-7], that is flexible enough to run with many types of protocols and multimedia formats, as MP3, MIDI for audio, MPEG-4 for video and JPEG, PNG, BMP, SVG for pictures. The MMAPI framework has been designed to run on any J2ME-based virtual machine, including the CDC and CLDC virtual machines. This standard Java specification has been defined by the Java Community Process (JCP) in JSR 135, [7].

At the core of the MMAPI package there are two concepts that describe multimedia processing:

* Protocol handling refers to the process of reading data from a source, as local files, from a streaming server or from device components, into a media-processing system;

* Content handling refers to the process of decoding the media data and rendering it to an output interface.

Figure 3 describes the three high-level object types, Data Source, Player and Manager, that the API provides, in order to implement the two concepts.

The relation between these components is based on the fact that the Manager object is used to create Player entities from DataSources or Input Streams. Once created, Player class provides methods that will give control over the multimedia resource.

2.1 Audio

Different audio formats are managed by the MMAPI using:

* Manager playTone() method that generates simple tones, characterized by frequency and duration; this is the simple audio media format that can be managed and its role is to provide basic audio capabilities to mobile devices; the following sequence play a tone for 2000 milliseconds at volume 50

* Player object, created to access audio files stored on the device or on an Web server; the next code sequence gets an MP3 file over the network and plays it:

* InputStreams and Player classes to access device stored audio resources, as MP3 files; media can be stored as a resource within the JAR file or in a RecordStore:

2.2 Graphics and Animation

The concept of graphics and animation management is direct related to the framework elements that allow developers to control device display. One core element of the J2ME platform used in this scope is the Canvas class, [9], defined in the javax.microedition.lcdui package, that lets an application draw screens using the low-level user-interface API.

The next code sequence is used to draw an image on the device display, creating a splash screen.

The image is load from the JAR and figure 4 shows the result.

The limitations of Canvas class is generated by a lack of control over when a canvas repaints itself or over how quickly key and pointer events are received by the object. In order to overcome this drawback and to provide support for software gaming industry, it was designed the GameCanvas class, that extends the Canvas class.

The role of the GameCanvas class is to provide efficient resource management for animation and event handling. Basic elements of an animation and their classes are:

* animation background implemented by the TiledLayer class; the background is viewed as a grid with custom dimensions; this provides an efficient and simple way to manage interactions between game components and dynamic generation of the environment; based on few resources, as the background elements in figure 5, resources are multiplied and put in the grid; the following code sequence, creates a 6x7 grid and populates it with 32x32 pixels graphical elements, that were extracted from a PNG image

* animated elements, defined as Sprite objects; this class is designed to manage animation sequences independently from the main thread; the next sequence creates a Sprite object based on butterfly.png, described in figure 5; the sprite animates the butterfly by changing in a predefined sequence the four images;

* LayerManager class that provides game developer methods to manage animation elements; usign a LayerManager object, animation elements are interconnected and it is decided how and in which context they are made visible; next sequence puts together, in a GameCanvas, both the background and the sprite;

On advanced mobile devices that incorporate a camera, MMAPI includes support for accessing it from code. For that the player is created using a capture://video locator. By default, the image format used to store the camera capture is PNG. The next code sequence allows the user to capture an image by camera:

The key elements from the above sequence are:

* create Player object

...

* Manager.createPlayer("capture://video");

...

* getControl over the resource

...

* getSnapShot to get the captured image.

...

2.3 Video

J2ME provides video management with the VideoControl interface. A reference to the video resource is provided by the Player class and its getControl() method.

The below sequence plays a video file on mobile device:

3 Windows Mobile

DirectShow is available starting with Windows Mobile 5. This is technology is based on COM. It is highly recommended to play the media files in a separate thread.

3.1 Audio

Audio files can be played using:

* standard API function,

* a shell function (Windows Mobile 6 and up only)

* DirectShow

* third-party API

The PlaySound function is limited to PCM WAV formats. This call plays the specified file. If an error occurs, bResult will be set to FALSE:

For other formats, SndPlaySync function can be used. This function is related to Windows Mobile 6 shell. A sample call of this function is:

DirectShow can be used to play audio files. Depending on installed codecs, various formats can be played. The following code is used to play an audio file using DirectShow:

The thread waits until the audio file ends so that this code needs to be in a separate thread.

There is a third-party API developed that allows playing sound.

3.2 Graphics and Animation

The basic drawing primitives are provided by the GDI. Usually the drawing is handled on WM_PAINT message.

The following listing demonstrates how to use them to draw shapes and text using GDI API:

There is support for 3D graphics using Direct3D Mobile. There are limitations of Direct3D Mobile compared to Direct3D due to hardware constraints.

First, a device has to be initialized and its properties to be set. The device will be used to render the 3D graphics based on objects properties (coordinates and color). The distance to the objects, the view angle, the light, the view window and other properties can be controlled. The following code initializes the device and the pyramid that will be rendered:

The code that draws and rotates the pyramid is:

Figure 6 shows the capture of a mobile device screen running the Direct3D Mobile code presented in this section.

3.3 Video

Video can be played on Windows Mobile based devices using DirectShow or Windows Media Player control. These methods require the use of COM technologies.

The code is similar to audio playback using DirectShow, for video playback a window is required. The following code initializes the filter graph manager and the player window and plays a video file in full screen:

The video playback has to take place in a separate thread in order to keep the user interface functionally.

4 Symbian OS

4.1 Audio

The Symbian audio component allows playing and recording audio files, depending on installed codecs.

Audio files can be played using CMdaAudioPlayerUtility class. Applications that use it need a helper class that implements MMdaAudioPlayerCallback interface in order to receive notification about the player. Audio streams can be played using the CMdaAudioOutputStream class.

The following code implements a player class that can be used for audio files:

The audio file can be stopped, paused and resumed. Also it is possible to position in the audio file. The following code is used to play an audio file using the class above:

4.2 Graphics and Animation

Drawing is similar to other programming platforms. It is necessary a device context that will be used to draw on the display. The following code is used to draw some basic shapes and text on the application main window:

Usually this code is called from the Draw function, called when the screen needs to be painted.

Figure 7 depicts s a capture of a Symbian phone screen showing the result of running the code above.

Also multimedia framework has support for image processing.

4.3 Video

Symbian's multimedia framework has support for playing and recording videos. The source can be a file on disk or a stream of data. The architecture is similar to audio playback, an observer class need to be created to receive the framework events. Additionally a window is needed to playback the video. CVideoPlayerUtility is the class used for video playback and the interface MVideoPlayerUtilityObserver is required to handle the playback events through callback functions.

The following class can be used to play video files:

In order to use the video player class, an object needs to be initialized and to call the PlayL method:

The player objects need to be released when finished.

The camera can be used for video playback and recording, if available.

6 Conclusions

Using operating system native API leads to faster applications but long development curves and difficulties on porting applications to other operating systems.

Acknowledgement

This paper presents some results of the research project IDEI 2673: Project management methodologies for the development of mobile applications in the educational system, financed within the framework of IDEI research program.

References

References

[1] P. Pocatilu and A. Pocovnicu, "Multimedia for mobile Learning," Proc. of Informatics in Economy Conference, May 2009, pp. 362-367

[2] Adi Rome et al. Multimedia on Symbian OS - Inside the Convergence Device, John Wiley & Sons Ltd, 2008, p. 23

[3] Windows Mobile 5.0 SDK: Multimedia and Games [Online]. Available: http://msdn.microsoft.com/en-us/library/aa454194.aspx (March 2009).

[4] iPhone Reference Library [Online]. Available: http://developer.apple.com/iphone/library/navigation/index.html. (March 2009).

[5] The Developer's Guide|Android Developers [Online]. Available: (http://developer.android.com/guide/index.html (March 2009).

[6] Qusay Mahmoud - The J2ME Mobile Media API, Sun Developer Network, 2003, http://developers.sun.com/mobility/midp/articles/mmapioverview/

[7] JSRs: Java Specification Requests - JSR 135: Mobile Media API, Final release 3, 2006, http://jcp.org/en/jsr/detail?id=135

[8] Bruce Hopkins - Comparing Mobile Platforms: Java ME and Adobe Flash Lite, Sun Mobility Tech Tips, http://blogs.sun.com/mobility_techtips/entry/comparing_mobile_platforms_java_me

[9] Eric Giguere - Game Canvas Basics, Sun Developer Network, 2004, http://developers.sun.com/mobility/midp/ttips/gamecanvas/index.html