Archive
HowTo: Upgrade Kinect Audio Positioning code from older Beta SDK
Exploring the Kinect SDK v1.8 Developer Toolkit Browser application, I noticed some audio positioning visualizations at various samples (specifically at Audio Basics, Audio Explorer and Kinect Explorer samples), but I kept on looking for some simpler/cleaner sample code to return audio positioning info that would be easier to reuse.
So I can across http://kinectaudioposition.codeplex.com that looked like a good candidate, since it was defining a reusable KinectMicArray class to return the BeamAngle and SoundSourceAngle’s of a KinectAudioSource. There was also some sample code on how to use that KinectMicArray class at http://channel9.msdn.com/coding4fun/kinect/Kinect-Audio-Positioning
However, after downloading it, I realized it was written (hadn’t seen the notice it had on its page) for an older Kinect Beta SDK version. So I decided to port that code to Kinect SDK v1.8 (which is the latest Kinect SDK version to support Kinect for Windows v1.0 and Kinect for Xbox 360 sensors, since the newer Kinect SDK supports KfW v2.0 and Kinect for Xbox One – in fact KfW v2.0 is now “merged” with the Kinect for Xbox One product via the use of an external adapter with the later to connect to a PC).
During that code porting to Kinect SDK v1.8 I noticed the following issues that had to be addressed. At the older Kinect Beta SDK:
- one could instantiate a KinectAudioSource class directly instead of having to get it from a started KinectSensor, plus the object was disposable so one could place it in "using" clause
- one could do audioSource.SystemMode = SystemMode.OptibeamArrayOnly (that property doesn’t seem to be available anymore to set)
- audioSource.BeamAngleChanged was called BeamChanged
- audioSource.SoundSourceAngleChanged probably didn’t exist (have seen older code use a BackgroundWorker and do polling)
- audioSource.SoundSourceAngleConfidence was called SoundSourcePositionConfidence
- audioSource.SoundSourceAngle was called SoundSourcePosition
- audioSource.BeamAngle and audioSource.SoundSourceAngle (then called SoundSourcePosition) were returning radians instead of degrees (so one usually had to convert to degrees using 180.0 * rad / Math.PI)
I took the chance to cleanup and refactor the original code and XAML and since the original codeplex site just had a single download and not any source code in the repository to fork, I uploaded the whole thing ZIPed under their source code / patches area.
Gotcha: MediaElement Source=null releases stream, SetSource(null) fails
This is my contribution to:
If you use MediaElement, make sure you don’t get bitten by this one: http://msdn.microsoft.com/en-us/library/cc626563(v=vs.95).aspx
ArgumentNullException – The mediaStreamSource is null.
…After calling this method, MediaElement.Source returns null. If this is called and MediaElement.Source is set, the last operation wins.
If a MediaElement is removed from the UI tree while is has an opened MediaStreamSource, subsequent calls to SetSource may be ignored. To ensure featureSetSource calls will work, set the Source property to null before detaching the MediaElement from the UI tree.
naturally one would expect, if they only use SetSource(somestream) to use SetSource(null) to release the resources. Nope, they thought "better", you have to use Source=null instead to release resources and SetSource(null) throws ArgumentNullException
that is what I call a design bug (breaks the rule of "least expected" behavior and causes bugs that bite you at runtime only [unless somebody has made a static analysis rule to catch such a thing – would need metadata of course that some argument can’t be null, like in Code Contracts])
I managed to introduce this bug while refactoring some code in ClipFlair Studio‘s AudioRecorder control the other day 😦
Note that you can’t use at MediaElement something like Source = stream to open a Stream, since that is a Uri property (not an Object property to also accept Stream) and you have to use SetSource(stream) instead, so you’d also expect to be able to use SetSource(null) to release the resources.
Update: Fixed this in AudioRecorderView class (uses MVVM pattern) of AudioRecorderControl, at Audio property’s "set" accessor it needed the following null-guarding pattern:
if (mediaStreamSource != null)
player.SetSource(mediaStreamSource);
//must set the source once, not every time we play the same audio,
//else with Mp3MediaSource it will throw DRM error
else
player.Source = null;
Fix: Allow rewind of Mp3MediaStreamSource back to start
My comment at:
https://github.com/loarabia/ManagedMediaHelpers/issues/16
While trying to use Mp3MediaSource at ClipFlair Studio’s AudioRecorder control (http://clipflair.codeplex.com), I noticed that when I was doing Stop() at MediaElement and then Play() it kept on playing from where it was before at Mp3MediaStreamSource
So I did the following fix:
1) added these fields
/// <summary>
/// The first known frame (for rewind)
/// </summary>
private MpegFrame firstFrame;
/// <summary>
/// The first audio stream position (for rewind)
/// </summary>
private long firstAudioStreamPosition;
2) added to the end of ReadPastId3v2TagsCallback:
this.firstFrame = mpegLayer3Frame; //keeping the 1st frame position for rewinding
this.firstAudioStreamPosition = audioStream.Position;
3) changed SeekAsync:
/// <summary>
/// <para>
/// Only supporting seeking back to start of the stream
/// (e.g. when doing MediaElement.Stop()).
/// </para>
/// <para>
/// In a fuller MediaStreamSource, the logic here would be to actually seek to
/// the correct mpeg frame matching the seekToTime passed in.
/// </para>
/// </summary>
/// <param name="seekToTime">
/// The time to seek to (in 100-nanosecond units [hns])
/// </param>
protected override void SeekAsync(long seekToTime)
{
/*
if (seekToTime > this.trackDuration.TotalMilliseconds * 10)
{
throw new InvalidOperationException(
"The seek position is beyond the length of the stream");
}
*/
if (seekToTime != 0) //only supporting rewinding back to start
throw new InvalidOperationException(
"Only supporting seeking back to start of the stream");
else
{
this.currentFrame = firstFrame;
this.currentFrameStartPosition = MpegFrame.FrameHeaderSize;
this.audioStream.Position = firstAudioStreamPosition;
}
this.ReportSeekCompleted(seekToTime);
}
note that I changed the documentation for that method to say that time passed to it is in 100 ns units, not in ns units (according to WaveMediaStreamSource code that I found, unless that one has it wrong)
Fix: Silverlight Media Framework Player VolumeElement out of sync
I was just adding a storable (persistent) Volume property to the MediaPlayerView class used at ClipFlair’s MediaPlayerWindow (connected to the underlying SMF player’s VolumeLevel property), when I realized that after reloading saved state, the SMF player’s Volume control would show a different value than the value set to it (which I could confirm by flipping the MediaPlayer control and looking at its properties on the backpanel that the ClipFlair app features for each component and for the activity container itself).
It seems to be some bug in the SMF logic or in its default template, since the following fix makes the issue disappear. The trick is that at a descendent class from SMFPlayer (like the MediaPlayer class that ClipFlair’s MediaPlayerWindow uses), one can override the OnApplyTemplate method (a standard method in templated XAML-based controls at Silverlight and WPF) and call the UpdateVolumeElement method shown below.
That method temporarily sets the volume to another value than its current one (this is important, just setting again to the same value would be ignored), then set again the current value to force the VolumeElement control’s UI to update.
//Project: ClipFlair (http://ClipFlair.codeplex.com) //Filename: MediaPlayer.cs //Version: 20130211 using System; using System.Linq; using System.Windows; using System.Windows.Media; using Microsoft.SilverlightMediaFramework.Core; using Microsoft.SilverlightMediaFramework.Core.Media; using Microsoft.SilverlightMediaFramework.Core.Accessibility.Captions; using Microsoft.SilverlightMediaFramework.Plugins.Primitives; namespace ClipFlair.MediaPlayer { public class MediaPlayer : SMFPlayer { public override void OnApplyTemplate() { base.OnApplyTemplate(); //... UpdateVolumeElement(); //patch for SMF bug } protected void UpdateVolumeElement() { //patch for SMF to update VolumeElement UI with any already set VolumeLevel double volume = VolumeLevel; VolumeLevel = (volume == 1) ? 0.9 : 1; VolumeLevel = volume; } //...
Update: While submitting this as a bug to SMF source site on Codeplex it came to me that I could try setting the VolumeElement’s VolumeLevel value directly at ApplyTemplate and indeed it works:
protected void UpdateVolumeElement()
{
//patch for SMF to update VolumeElement UI with any already set VolumeLevel
VolumeElement.VolumeLevel = VolumeLevel;
}HowTo: Hide Video area of Silverlight Media Framework Player
Since our onoing project ClipFlair is a follow-up of the succesful project LeViS, during its development I’m also looking at covering possible use-cases I infer from feedback entries at LvS application issue tracker on Codeplex (note that ClipFlair is also opensourced on Codeplex at http://ClipFlair.codeplex.com).
One such issue (http://lvs.codeplex.com/workitem/11511) was titled “Allow video hiding while controller is visible for audio only”, so since Silverlight Media Framework (SMF) is used for the media player in ClipFlair, I looked into how to tell it to hide the video area.
Well, it seems that to hide the video and hear just the audio in SMFPlayer one needs to set MediaPresenterElement‘s (a property of SMFPlayer class) MaxWidth and MaxHeight to 0. One can do that at an overriden OnApplyTemplate method in a class descding from SMFPlayer, or could also do it at a custom SMF player template.
Note that setting MediaPresenterElement’s Visibility property to Visibility.Collapsed makes the player not load the video at all, so that one was not an option.
One could define a VideoVisible DependencyProperty at a class descending from SMFPlayer:
#region VideoVisible /// <summary> /// VideoVisible Dependency Property /// </summary> public static readonly DependencyProperty VideoVisibleProperty = DependencyProperty.Register("VideoVisible", typeof(bool),
typeof(MediaPlayer), new FrameworkPropertyMetadata(false, FrameworkPropertyMetadataOptions.None, new PropertyChangedCallback(OnVideoVisibleChanged))); /// <summary> /// Gets or sets the VideoVisible property. /// </summary> public bool VideoVisible { get { return (bool)GetValue(VideoVisibleProperty); } set { SetValue(VideoVisibleProperty, value); } } /// <summary> /// Handles changes to the VideoVisible property. /// </summary> private static void OnVideoVisibleChanged(DependencyObject d,
DependencyPropertyChangedEventArgs e) { MediaPlayer target = (MediaPlayer)d; bool oldVideoVisible = (bool)e.OldValue; bool newVideoVisible = target.VideoVisible; target.OnVideoVisibleChanged(oldVideoVisible, newVideoVisible); } /// <summary> /// Provides derived classes an opportunity to handle changes to the
/// VideoVisible property. /// </summary> protected virtual void OnVideoVisibleChanged(bool oldVideoVisible,
bool newVideoVisible) { MediaPresenterElement.MaxWidth = (newVideoVisible)?
double.PositiveInfinity : 0; MediaPresenterElement.MaxHeight = (newVideoVisible) ?
double.PositiveInfinity : 0; } #endregion
At first, to restore the MaxWidth/MaxHeight to show the video area again I tried to set them to double.NaN which didn’t work. Then I checked MaxWidth documentation which was saying:
The maximum width of the element, in device-independent units (1/96th inch per unit). The default value is PositiveInfinity. This value can be any value equal to or greater than 0.0. PositiveInfinity is also valid.
and
String representation of a Double value equal to or greater than 0.0. This is interpreted as a device-independent unit (1/96th inch) measurement. Strings need not explicitly include decimal points. For instance a value of 1 is acceptable.
The same Double range restrictions as mentioned in the Property Value section apply, except that you must use x:Static Markup Extension to set the value to be PositiveInfinity.
Per the 1st quote, one just needs to set MaxWidth or MaxHeight to double.PositiveInfinity (and not double.NaN that I originally expected) to reset it to its default value, that is behave as if a maximum width or height respectively has never been set.
As for the second quote, it basically says that XAML double fields need special treatment in order to specify a positive infinity value (if you ever need to, since you’d usually just skip the MaxWidth/MaxHeight field, unless you need to override some non-default inherited value). You would need to define a XAML namespace to point to the System namespace at mscorlib assembly (library), either at the root control in the XAML or directly where you need it:
<MyNameSpace:MyControlxmlns:sys="clr-namespace:System;assembly=mscorlib"
MaxWidth=”{x:Static sys:Double.PositiveInfinity}”
MaxHeight=”{x:Static sys:Double.PositiveInfinity}”
…
/>
BTW, to hide the controller bar (called ControlStrip in SMF) you just set IsControlStripVisible property to false.
HowTo: Compress speech audio using CSpeex codec at Silverlight apps
Following up on my post on how to fix CSpeex (http://cspeex.codeplex.com) to compile at Silverlight 5 (see https://zoomicon.wordpress.com/2012/06/09/fix-compile-cspeex-audio-codec-at-silverlight-4-rc-and-silverlight-5/).
Initially I thought it was not working correctly, but then I noticed I was running it on Terminal Server and didn’t have an audio device available. Running SLAudioDemo included in the CSpeex distribution (after doing the fix mentioned at my post above) seems to work fine on my local PC.
The SLAudioDemo demonstrates capturing audio into memory as a compressed CSpeex stream which it can decompress and save into a .WAV file that one can play say with Windows Media Player. The WAV is uncompressed, since SLAudioDemo compresses in memory (via StreamAudioSink) and decompresses to normal PCM WAV when you save (else it would be unplayable).
Obviously the Speex codec is useful for uploading (e.g. via HSS Interlink) compressed audio to a server (or to the other end at some voice chat app) and decompress there before playback (similarly the other way around).
At the server side if you want to do audio storage too, you could also keep an extra decoded copy of the compressed Speex audio data, recoded into WMA using automation of Microsoft Expression Encoder for example (see Microsoft Transform Manager, can do this in the background if you uncompress into a .WAV file at some incoming folder). That way you could serve that audio to your Silverlight client as Speex-encoded data file, but also to other users that use Windows Media Player to get the audio file from some URL etc. Could also encode to MP4/AAC or MP3 but there are licensing issues with those from what I know.
For the CSpeex encoder see StreamAudioSink.cs. Only 2 lines are needed in MainPage.xaml.cs of SLAudioDemo:
streamAudioSink = new StreamAudioSink();
streamAudioSink.CaptureSource = _captureSource;
after you 1st declare at the top level (must have this as a top-level field, else the garbage collector can get it)
private StreamAudioSink streamAudioSink;
Here’s the decoding code that saves the WAV (from MainPage.xaml.cs):
if (sfd.ShowDialog() == true)
{
// User selected item. Only property we can get to is.
using (Stream stream = sfd.OpenFile())
{
JSpeexDec decoder = new JSpeexDec();
decoder.setDestFormat(JSpeexDec.FILE_FORMAT_WAVE);
decoder.setStereo(true);
Stream memStream = streamAudioSink.MemFile.InnerStream;
memStream.Position = 0;
decoder.decode(new RandomInputStream(memStream),
new RandomOutputStream(stream));
stream.Close();
}
}
Don’t get puzzled by the use of RandomInputStream / RandomOutputStream. These are from Java’s I/O package (java.io.*) that CSpeex has ported to .NET, since it’s in fact a port of JSpeex, a Java implementation of the Speex codec.
Fix: compile CSpeex audio codec at Silverlight 4 RC+ and Silverlight 5
Just added an issue to CSpeex project – http://cspeex.codeplex.com/workitem/18209 – with some fixes to make it compile for Silverlight 4 RC and higher (was for Silverlight 4 Beta).
1) at "MainPage.xaml.cs", replace with this method implementation:
private void TakeSnapshot_Click(object sender, RoutedEventArgs e)
{
if (_captureSource != null)
{
// capture the current frame and add it to our observable collection
_captureSource.CaptureImageCompleted += (source, eventargs) =>
{
_images.Add(eventargs.Result);
};//”ImageCaptureAsync" method replaced by "CaptureImageAsync" at Silverlight 4 RC
_captureSource.CaptureImageAsync();
}
}
2) at "StreamAudioSync.cs" do this small change:
protected override void OnFormatChange(AudioFormat audioFormat)
{
if (audioFormat.WaveFormat == WaveFormatType.Pcm)//”PCM" renamed to "Pcm" at Silverlight 4 RC (after SL4beta)
{
…
Note that the CSpeex-SL project won’t load if you don’t have Silverlight 3 SDK installed. After loading you can open up properties of that project and of SLAudioDemo one and set the target to Silverlight 5 from Silverlight 3, then save those two projects and rebuild the solution.
George Birbilis @zoomicon 