HowTo: Multi-row captions and wrapping at Silverlight Media Framework

One of the main goals of the ClipFlair project is to explore the use of Video Captioning (together with Revoicing) for Foreign Language Learning (FLL). There both are important enhancements compared to the simpler two-line subtitling that was put in use in its ancestor project LeViS.

Captioning means that multi-row text is needed while text wrapping is also useful, especially if cartoon-like balloons over the video area are to also be examined as an option. Implementing multi-row captions and caption wrapping was a bit tricky in the SMF based player, due to the lack of relevant documentation, so adding some relevant notes below.

As I discuss in more detail near the end of the discussiion at  http://smf.codeplex.com/discussions/245424, the way I finally managed to make SMF player show multi-row captions is by setting the caption region bounds (via Origin and Extend properties of the CaptionRegion object) to take up nearly all the video area and combined it with DisplayAlign = DisplayAlign.After (to bottom align the caption rows group) and Overflow = Overflow.Dynamic (to extend the visible caption area dynamically inside the defined bounds).

Regarding caption wrapping there seems to be a bug in SMF, it wraps caption text at the video boundary instead of at the caption region max boundary (as defined by Origin/Extend properties of the CaptionRegion object). Have reported it at:

http://smf.codeplex.com/workitem/23634

A workarround for caption wrapping to work perfectly is to set top to 0 for the origin and set extend width to 100% for the caption region, but this won’t be useful if you want to show cartoon-like bubbles on the video for some captions. If Microsoft don’t fix that in the near future I guess I’ll have to fix it myself at the source-code of SMF (luckicly it’s an open-source project).

The relevant code snippet  from http://ClipFlair.codeplex.com for the MediaPlayer component is:

private const double CAPTION_REGION_LEFT = 0; //0.05;
private const double CAPTION_REGION_TOP = 0.05;
private const double CAPTION_REGION_WIDTH = 1; //0.9; //SMF 2.7 has a bug here,
               //it wraps caption text at the video boundary instead of at the 
               //caption region max boundary (as defined by Origin and Extend)
private const double CAPTION_REGION_HEIGHT = 0.9;

public static void StyleCaptions(CaptionRegion theCaptions)
{
  if (theCaptions == null) return;

    theCaptions.Style.ShowBackground = ShowBackground.WhenActive; 
               //doesn't seem to work if other than transparent color is used
    theCaptions.Style.BackgroundColor = Colors.Transparent;

    //set caption region (max) bounds
    theCaptions.Style.Origin = new Origin() { 
      Left = new Length() { 
        Unit = LengthUnit.Percent, Value = CAPTION_REGION_LEFT }, 
      Top = new Length() {
        Unit = LengthUnit.Percent, Value = CAPTION_REGION_TOP } };
    theCaptions.Style.Extent = new Extent() { 
      Width = new Length() { 
        Unit = LengthUnit.Percent, Value = CAPTION_REGION_WIDTH }, 
      Height = new Length() { 
        Unit = LengthUnit.Percent, Value = CAPTION_REGION_HEIGHT } };

    //theCaptions.Style.Direction = Direction.LeftToRight;
  
    theCaptions.Style.DisplayAlign = DisplayAlign.After; 
                                 //align multirow catpions to bottom of region
    theCaptions.Style.TextAlign = TextAlignment.Justify; 
                                 //horizontally center captions

    theCaptions.Style.WrapOption = TextWrapping.Wrap; 
                                 //wrap too long captions to next row
    theCaptions.Style.Overflow = Overflow.Dynamic; 
         //extends the area for the captions as needed, up to the given Extent

    foreach (CaptionElement caption in theCaptions.Children)
      StyleCaption(caption);
}

public static void StyleCaption(TimedTextElement theCaption)
{
  if (theCaption == null) return;

  theCaption.CaptionElementType = TimedTextElementType.Text;
  theCaption.Style.ShowBackground = ShowBackground.WhenActive;
  theCaption.Style.BackgroundColor = Color.FromArgb(100, 0, 0, 0); 
                                     //use a semi-transparent background
  theCaption.Style.Color = Colors.White;

  Length length = new Length
 {
   Unit = LengthUnit.Pixel, //must use this, since the default LengthUnit.Cell
                           //used at TimedTextStyle constructor is not supported
   Value = 20
 };
  theCaption.Style.FontSize = length;
}

I also tried setting Padding property at the code above in case it would help with the correct wrapping, but it somehow results in crashing Silverlight (may be related to the fact that I have hardware acceleration enabled for our Silverlight app).

LvS (Learning via Subtitling) opensource VB.NET application for FLL learning

The LvS application for Foreign Language Learning (FLL) via Film Sutitling that I co-designed and originally authored in Visual Basic.net 2005 (latest version was compiled on VB.net 2008) for Hellenic Open University in the context of EU-funded research project LeViS, is opensourced at CodePlex (Microsoft’s community site for opensource projects):

http://www.codeplex.com/LvS

Project Description

Learning via Subtitling: Software & Processes for Developing Language Learning Material based on Film Subtitling (http://levis.cti.gr)

The main focus of Learning Via Subtitling is the development of educational material for active foreign language learning based on film subtitling. It aims to cover the exigency for active learning where cultural elements are involved effectively through real-life (simulated) activities and the need for productive use of multimedia not as nice add-on but as the core of an activity. LvS is a software environment designed and implemented to serve the pedagogical needs of the LeViS project.

Funding framework: Socrates Programme, LINGUA 2 (Development of Language Tools and Materials), European Commission, DG for Education and Culture

Coordinating Institution: Hellenic Open University (HOU), http://www.eap.gr

Description of the LvS Environment

The LvS environment
Learning via Subtitling (LvS) is a subtitling simulator designed for educational activities’ purposes. This software tool can be used by language teachers to create activities based on subtitling film-scenes, news, documentaries etc., and by FL learners to carry out these activities.

It comprises four basic areas – Multimedia player, Document viewer, Subtitle editor and Notes – as shown in the screenshot below.

Screenshot of an LvS learning activity

Multimedia player area
It allows the learner to view, rewind and forward the film, both with and without subtitles.

Document viewer area
It allows the learner to view the instructions and other files necessary for the activity (information about the clip, the script, exercises, etc.)

Subtitle editor area
It allows the learner to edit and manage the subtitles. Each subtitle line is divided in four columns where the subtitle’s data is viewed: Start time and End time (the temporal points in the clip when the subtitle text appears and disappears from the screen), Duration, and Subtitle text. The next two columns are for teacher and learner comments. The teacher can mark the subtitle line with an icon (“well done”, “warning” etc.) which when clicked takes the learners to the Notes area where they can read the teacher’s comment.

Moreover, an algorithm calculates the number of characters introduced in each subtitle: if it exceeds the time and space constraints, the text is marked. This does not affect the appearance of the subtitle on the multimedia player and the subtitle file can still be used and saved.

Also, if the number of the subtitle (first column) is double-clicked, the “current time” of the clip is moved to the equivalent time-point, enabling the learner to view and listen only the fragment where the selected subtitle appears.

The users can create new subtitles in the subtitle file by clicking on the buttons “Start subrtitle” and “End subtitle” in order to establish the time when they want the subtitle to appear and disappear respectively. This process is called “cueing”. Whether the cueing is performed by the learner or the teacher depends on the design and the objectives of the activity. If the teacher aims to reduce the technicalities of subtitling, s/he may provide the in and out times of the subtitles together with the rest of the activity elements. On the other hand, if the objective is for the students to practice listening, this process may be carried out the students: in order for them to find when exactly a new subtitle should appear, they are obliged to listen to the clip over and over again in order to insert the in and out times correctly.

Notes area
It allows the learner and the teacher to exchange feedback. It is divided in the general notes and the comments per subtitle. When a comment per subtitle is clicked, the “current time” of the clip is moved to the moment when the respective subtitle appears.

Other Software Features:

• Authoring mode (see Glossary in this tutorial)
• Importing (Packed activity, Video, Subtitles, Documents)
• Exporting (Packed activity with or without video, Subtitles, Documents)
• Student and Teacher general Notes and per subtitle Comments
• Multiple Documents in different tabs available
• Auto-resizing parts of the interface
• Option to write subtitle text directly under the player with simultaneous update of the subtitle grid
• Most recently used files list
• Buttons “Set subtitle start” and “Set subtitle end” modifying the Start time and End time of an existing subtitle

The files needed for the creation of an activity (input files) are:

a) a multimedia file: eg. a film scene, a video-clip, a documentary.
b) one or more documents: e.g. a powepoint presentation with the instructions, an MS Word document with an exercise.
c) a subtitle file – unless the activity requires the students to create their own subtitles.

LvS runs under Windows 2000, Windows XP, Windows Vista and Windows 7 (should run ok on Windows Server 2003 and 2008 too I guess)