Friday, September 26, 2008

Fullscreen video in gstreamer with gtk+

UPDATE: Source files are now up on gitorious.

In a similar vein to my previous entry, I'd like to explain how to get full screen video with gstreamer and gtk. Here again I'm using the latest CVS head of gstreamer, as well as the gtk+ and gdk development files.

I was very impressed with gtk's well documented API and in-depth tutorial. Also, since both gstreamer and gtk depend on GObject, combining functionality from both frameworks is quite seamless.

The source file fullscreen.c and accompanying Makefile are used for this example.

To test that your build environment is properly setup, I would recommend making a simple C file with the following:

#include <gst/gst.h>
#include <gtk/gtk.h>
#include <gst/interfaces/xoverlay.h>
#include <gdk/gdk.h>
#include <gdk/gdkx.h>

gint main(gint argc, gchar *argv[])
{
gst_init(&argc, &argv);
gtk_init(&argc, &argv);
return 0;
}


and try compiling. Make sure that you have all the necessary development files installed and that your environment can find them. The gst_init and gtk_init calls must always be called once, by any program using gstreamer and gtk. Note that if for some reason they are called more than once, the calls have no effect.

To be able to process key-events and to keep the pipeline rolling, we need to use glib's mainloop. It may be possible to achieve the same results with some other event loop mechanism, this is just the one most often used in gtk and gstreamer applications.

loop = g_main_loop_new (NULL, FALSE);

The simple pipeline here consists of a videotestsrc going to an xvimagesink. We set the "force-aspect-ratio" property of the xvimagesink to TRUE so that when the size is changed, the image's proportions are not distorted.

We build our gtk window:
window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
g_signal_connect(G_OBJECT(window), "expose-event", G_CALLBACK(expose_cb), sink);

and attach the "expose-event" to the expose callback function. This function will be called when our window is brought to the foreground. The expose callback overlays xvimagesink's video on our gtk window.

A common feature in video-players is to assign a hot key to switch from windowed to full screen viewing. This is possible using:

gtk_widget_set_events(window, GDK_KEY_PRESS_MASK);
g_signal_connect(G_OBJECT(window), "key-press-event", G_CALLBACK(key_press_event_cb), sink);

which connects are key_press_event_cb function to the "key-press-event" signal emitted by the gtk window. The call to gtk_widget_set_events makes sure that the key-press event is propagated all the way up to our top-level window. This is important for determining which level of a window hierarchy is intended to handle such an event.

Lastly, we need to make the window black initially, as otherwise the background will sometimes be white when switching from fullscreen to windowed mode.
We call gtk_widget_show_all() on our window to "recursively show a widget, and any child widgets (if the widget is a container)". The pipeline is then set to playing, and the mainloop is run.


gstreamer test video

Thursday, August 21, 2008

Multichannel audio with gstreamer

Over the past few months, I have spent a lot of time working with the gstreamer multimedia framework. From their site:
GStreamer is a library for constructing [...] graphs of media-handling components. The use cases it covers range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.
I've found gstreamer to be remarkably flexible and useful for a variety of audio-video applications. One of the trickier things I had to figure out was how to have 8 channels of audio playing in one gstreamer pipeline.

These examples require the following packages:
  1. JACK Audio Connection Kit librairies
  2. gst-plugins-base-0.10.20
  3. gst-plugins-good-0.10.9
  4. gst-plugins-bad-0.10.8

These modules should be installed in the above order. Personally I use the CVS head for all of the above, which you can get by doing:
$ cvs -d:pserver:anoncvs@anoncvs.freedesktop.org:/cvs/gstreamer co modulename
where modulename is gstreamer, gst-plugins-base, gst-plugins-good, and gst-plugins-bad respectively.

Gstreamer includes a command-line utility, gst-launch, that allows a user to quickly build a gstreamer pipeline with a simple text description. For example:


$ gst-launch audiotestsrc ! jackaudiosink

will play a sine wave, provided you are already rolling a jack server. I generally run a jack server using the qjacktl application.

Users should be aware of this warning from the gst-launch man page:

gst-launch is primarily a debugging tool for developers and users. You should not build applications on top of it. For applications, use the gst_parse_launch() function of the GStreamer API as an easy way to construct pipelines from pipeline descriptions.

It is possible to run a simple multichannel audio example with the following launch line:

gst-launch-0.10 -v interleave name=i ! audioconvert ! audioresample ! queue ! jackaudiosink \
audiotestsrc volume=0.125 freq=200 ! audioconvert ! queue ! i. \
audiotestsrc volume=0.125 freq=300 ! audioconvert ! queue ! i. \
audiotestsrc volume=0.125 freq=400 ! audioconvert ! queue ! i. \
audiotestsrc volume=0.125 freq=500 ! audioconvert ! queue ! i. \
audiotestsrc volume=0.125 freq=600 ! audioconvert ! queue ! i. \
audiotestsrc volume=0.125 freq=700 ! audioconvert ! queue ! i. \
audiotestsrc volume=0.125 freq=800 ! audioconvert
! queue ! i. \
audiotestsrc volume=0.125 freq=900 ! audioconvert ! queue ! i.

This pipeline consists of 8 audiotestsrc elements, which generate sine tones of increasing frequency. The audioconvert element is used to convert a given audiotestsrc's output to the appropriate numeric data type that the queue element expects. The queue element is a simple data buffer, to which the audioconvert element writes, and from which the interleave element reads. The interleave element combines multiple channels of audio into one interleaved "frame" of audio. For example, if we had 2 independent channels of audio like so:

Channel1: 00000...
Channel2: 11111...

where channel 1 outputs only 0's, and channel only 1's, the interleaved frame would look like:

0101010101...

The interleaved audio again needs to go through an audioconvert and an audioresample element in case the audio from our pipeline differs in datatype or sample rate from the jack server. Finally the audio is output by the jackaudiosink element, which writes audio from our pipeline into corresponding jack ports.

Many plugins require that the interleave element explicitly specify each channel's spatial position. Unfortunately, this cannot be done with gst-launch. I've created an example C program, multiChannel.c, which initializes interleave appropriately. It can be compiled with this Makefile. The relevant section in multiChannel.c is the function set_channel_layout(GstElement *interleave). This function is passed our interleave element, and sets its channel-positions property to an array of valid spatial positions.

Gstreamer uses Glib's object system (GObject) heavily, and as a result the above example program might be a little tricky to follow for programmers used to straight C. Check out gstreamer's application development manual for further examples of gstreamer usage in C.

Monday, February 25, 2008

Metronome project

The ViMic software I worked on will be used in a piece on March 4th. I will post some notes about the current state of the project and how it was used after this concert.

Besides ViMic, I have been working on a customizable polymetric metronome built using the STK for a few months now, as a personal project. After looking at several GUI toolkits, I have decided to develop the interface for this program using wxWidgets. Since I have only used QT before (and only a little at that), I was unsure for quite a while about which toolkit would work for my project. I was mainly looking for something that was easy to develop crossplatform, in "straightforward" (aka does not feel like I'm learning another language) C/C++. I was persuaded by a friend to try wxWidgets, and was more or less sold by Jeff Cogswell's wxWidgets tutorial. Also the free wxWidgets ebook didn't hurt either.

I won't say too much more about the metronome at this point, I want to keep the feature list short for the time being. A few important points:

  • It will be licensed under the GPL version 3.
  • It will have customizable ticking sounds (pitch, resonance, loudness, etc.).
  • It will allow for multiple metres at the same time.
  • It will be easy to use.

The last feature is really the most important, but out of habit the GPL notice always comes first. Besides, mentioning ease of use last makes it more memorable (for me at least) than if I had listed it second.

Wednesday, January 9, 2008

Source directivity

Sound source directivity patterns are the newest (and possibly last, for a while) feature to be added to the ViMic project. The user will be able to load or draw a directivity pattern in a Max table object. The ViMicMax~ external will use this table to obtain a directivity gain value. It remains to be seen whether or not the source object should have its own copy of the table, or alternately use a built-in function to read individual values from the table.

Monday, January 7, 2008

Room model filtering

So I'm now working on a spatial audio project called ViMic, in C/C++ and Max/Msp. I will post a more comprehensive overview at some point, but I think I should write about the current issue I am working on. The problem is as follows:

  • To avoid the Doppler effect when moving a sound source rapidly (i.e. sudden change in delay time resulting in a pitch shift), the program crossfades between the previous delay time and the new delay time.
  • This approach is good for avoiding the Doppler effect, but has led to discontinuities in the signal. These result from reading from the wrong part of the delay line.
  • Adding filtering to simulate the walls absorption of the sound seems to worsen the discontinuities apparent in the crossfading case.

A potential solution we are working on right now is to have one set of filters for the signal that is faded out, and another for the signal that is faded in. These filter sets are being swapped to avoid discontinuities. Moving the filter swap to the start of the crossfade, rather than the end, seems to have resolved the issue.