Snapshots from a live video source in gstreamer

EDIT: A helpful reader pointed out that using videorate in the pipeline should do the trick, something like:

gst-launch -v v4l2src ! tee name=t ! queue ! xvimagesink t. 
! queue ! videorate ! video/x-raw-yuv, width=640, height=480, framerate=1/1 ! jpegenc ! 
filesink location=test.jpeg

I'm fairly sure I tried this before with no success, but it works fine now. There have been some major bugfixes to videorate since I wrote this. In any case, I'll leave the rest of the article up in the hope that it is still useful.

---

As part of the propulse[ART] project, we wanted to add a "preview" feature to the existing audio/video engine. This involves writing a frame of video every second to a jpeg file, while displaying the video at a full 30 fps in a window. I was surprised to discover that this feature was not already implemented in gstreamer for live sources (to the best of my knowledge).
This should probably be implemented by a real gstreamer element, but for our purposes a relatively straightforward hack prototyped in streamshot.cpp was sufficient. The video pipeline consists of a video4linux source, a capsfilter to enforce some properties on the video, an ffmpegcolorspace element, and an xvimagesink. We attach a callback to the source pad of the video4linux source that is called every time our source has a new buffer of data (i.e. a frame). Since timing is not a big concern, the cb_have_data callback knows to only write a file every second by checking a boolean value that is periodically set to true by a separate callback. The cb_have_data function makes a copy of the buffer, swaps its red bytes with its blue bytes (swapping the red_mask and blue_mask in the caps did not work for some reason) and writes the modified buffer to a jpeg, line-by-line. The jpeg writing code was based on Andrew White's Xlib screenshot example. Thanks also to Koya Charles for help debugging this example, as well as the byte swapping.
The final version of the preview feature will probably happen entirely in a separate thread and not involve file-writing, but rather be streamed as these frames will be part of our web-interface.


Update: To make up for this entry's lack of flashiness, enjoy this code_swarm video of the propulse[ART] software's (codename miville) development so far (code_swarm instructions courtesy of amix.dk):

Computer Vision on OS X with Python and OpenCV

After my macbook's sudden demise and spontaneous, inexplicable regeneration, I've decided to try and port some things I've worked on in GNU/Linux to OS X. Also, having a built-in microphone and camera on a portable computer is pretty amazing when you work primarily with audio and video and need to test things.

I was really happy with how my previous forays in pygame had been going, so over the holidays I thought I 'd try and make a version of my previous camLoops prototype, but for OS X.
Also, at Alexandre Quessy's invitation, I've added cam_loop.py to the ToonLoop project. More on that in the coming weeks, so stay "tooned" (if not put off by that terrible pun).

*UPDATE The initial opencv version of camLoops is now available. I've only tested it on a MacBook running OS X so far, but it should also work in GNU/Linux.

I did some research online and there did not seem to be much in the way of camera- frame-grabbing modules for OS X with python bindings (granted it is a pretty niche area). The most complete are Apple's QTKit, the Cocoa framework (which has python bindings) for Quicktime and OpenCV, or Open Source Computer Vision, which is a cross-platform, BSD-licensed library written in C with Python bindings. Since I have no interest in wasting time using a Mac-only, proprietary framework like QTKit, the choice was pretty obvious. The installation, however, was not.

Given my heavy use of fink packages (when I should really just give in and roll GNU/Linux on my laptop), I am used to having to fight with cross-platform librairies/frameworks. I was even pleased to find an OS X specific build instruction page on the OpenCV wiki. It's quite likely that my approach is not the best solution for building but it's what worked for me.

I decided for better or worse to get the current production version of Python, which is 2.6.1 at the time of this writing. I grabbed the disk image and ran the installer (and breathed a sigh of relief). For pygame to work, I had to get PyObjC, which is used to build Cocoa apps for OS X in Python. This is not required for OpenCV, so skip ahead if you're not using pygame. This also required an svn checkout of the 1.4 branch of pyobjc which works on Tiger, as no slick disk-images with installers are available from the website at present. To grab the branch, do:

svn co http://svn.red-bean.com/pyobjc/branches/pyobjc-1.4-branch pyobjc

Fortunately, the PyObjC people know their target audience and all it took once in the pyobjc directory to make a nice installer was:

python setup.py bdist_mpkg --open

*UPDATE Previously OpenCV was using CVS for version control, they have since migrated to Subversion. Check out a clean version with:

svn co https://opencvlibrary.svn.sourceforge.net/svnroot/opencvlibrary opencvlibrary
In the INSTALL file, it is suggested that you do autoreconf -i --force
This failed for me (even though my autotools are up to date) so i used the pre-existing configuration files and left autoconf alone.

In the opencv directory, create a build directory and enter it with mkdir build; cd build

From the build directory, run configure with a few flags set (replace sw with /opt/local if you are using darwinports instead of fink):

../configure CPPFLAGS="-I/sw/include" LDFLAGS="-L/sw/lib" --with-python

Now compile and install with make && sudo make install. You will be prompted for your password.

Edit your ~/.profile to include the following two lines, which will be different if you set the --prefix option on your configure script to something other than the default /usr:

LD_LIBRARY_PATH=/usr/local/lib:${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH
PYTHONPATH=/usr/local/lib/python2.6/site-packages:${PYTHONPATH}
export PYTHONPATH
To test that everything worked, run your python interpreter and try import opencv. Provided that there is no errors such as 'no module opencv', you can start trying the python examples in the samples directory.


I ran python facedetect.py 0, where 0 is the index of the camera you want the program to use. The program then draws a red square around what it thinks is your face. Extra points for being beard proof, and hats off to Mark Asbach for his work on OS X support for OpenCV.

Live animation with pygame and video4linux

 EDIT: I've uploaded the code (and merged it into one file) on gitorious.

Last Friday, the SAT saw the premiere of a new work by Alexandre Quessy and Isabelle Caron called Motifs Urbains. The work was part of the experimentation phase of the propulse[art] project. The demo involved Alexandre generating live, multichannel audio and looping it, while at the same time in a separate room Isabelle did live stop-motion animation and looped it using Alexandre's Processing-based software, ToonLoop. Both rooms featured simultaneous playback of the audio and animation.
Inspired by the demo and my recent tinkering with pygame, I decided to try and write a prototype for a simple stop motion animation looper like ToonLoop. The result is cam_loop.py. Thanks to Nirav Patel's 2008 Google Summer of Code project, pygame (subversion revision 1744 or later) now supports video4linux camera input. The camLoop program shows the camera playback in the left hand region of the window, and the animation in the right hand of the video. The user can then grab frames live, and the accumulated frames will be played back in the right hand side of the window:

If you run the program during the winter months, you'll be subjected to a nauseating holiday theme overlayed on the incoming video. This example was simply to demonstrate how it's possible to easily draw arbitrary objects on live video. From Nirav's blog, it would appear that CoreVideo support is also forthcoming.