Optimized GStreamer media plugin for OMAP35x

[debernardis]

"Aiming to improve Linux multimedia development using its processors, Texas Instruments (TI) has released a version of the open source GStreamer media plugin for its OMAP35x and DaVinci system-on-chips (SoCs). The Linux-based GStreamer library supports development of audio and video playback, audio and video synchronization, and recording capabilities, says TI."

Could this give more oxygen to multimedia apps on our tablets?

http://www.linuxdevices.com/news/NS7463276436.html?kc=rss

https://gstreamer.ti.com/gf/project/gstreamer_ti/?DCMP=DSP_Gstreamer_5_18&HQS=Other+OT+gstreamer-pr

[qole]

Sadly, that will do nothing for the current-gen tablets. All that stuff is obviously aimed squarely at the next-gen Maemo 5 devices, which use the OMAP3 SoC, rather than the OMAP2 SoC on our beloved tablets.

The good news is that we will probably be able to use GStreamer to do A/V tasks on the new tablets. This means recording audio and video, streaming A/V out to the Net, etc. All with very little programming.

EDIT: With the new devices having a lot of processing muscle, and GStreamer plugins already written for them, there are some very interesting options opening up, such as a mobile media server (http://packages.ubuntu.com/jaunty/flumotion)...

[qole]

I'm getting more and more excited as I read about this Flumotion server (http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/)... Looks pretty straightforward to set up a web cam to stream out to the web (http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/help-configuration-assistant-producer-video-webcam.html)...

Makes me wonder if this can be done already using the tablet as the "Video Worker (http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/section-architecture-managers-and-workers.html)" and a desktop computer as the "Manager," "Muxer Worker," and "Streamer Worker". That might take some of the strain off of the poor little tablet.

http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/figures/diagram-architecture-with-workers.png

[Baloo]

Clearly the next tablet is going to be heavily orientated around its media capabilities. At OSiM last year there was a lot of talk about the camera and what you could do with it, video, photo's e.t.c. I can't see why streaming wouldn't be a big factor too. With the online OVI presence Nokia has, uploading videos and even live streaming would fit in with their slogan "Share photos and videos. Anytime. Anywhere."[1]

[1] http://share.ovi.com/?utm_campaign=OviShare&lid=ShareBridge

[lma]

Sadly, that will do nothing for the current-gen tablets. All that stuff is obviously aimed squarely at the next-gen Maemo 5 devices, which use the OMAP3 SoC, rather than the OMAP2 SoC on our beloved tablets.

I wonder how different the OMAP3 DSP is - maybe some of these codecs could be ported?

gst-ti-plugin-full-0.99.00/ti_build/ticodecplugin/src/gstticodecs_omap3530.c lists the following:


/* Audio Decoders */
{
"AAC Audio Decoder", /* String name of codec used by plugin */
"aachedec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
}, {
"MPEG1L2 Audio Decoder", /* String name of codec used by plugin */
"mp3dec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
}, {
"MPEG1L3 Audio Decoder", /* String name of codec used by plugin */
"mp3dec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
}, {
"WMA Audio Decoder", /* String name of codec used by plugin */
"wmadec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
},

/* Video Decoders */
{
"H.264 Video Decoder", /* String name of codec used by plugin */
"h264dec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
}, {
"MPEG4 Video Decoder", /* String name of codec used by plugin */
"mpeg4dec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
}, {
"MPEG2 Video Decoder", /* String name of codec used by plugin */
"mpeg2dec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
},

/* Image Decoders */
{
"JPEG Image Decoder", /* String name of codec used by plugin */
"jpegdec", /* String name of codec used by CE */
decodeEngine /* Engine that contains this codec */
},

/* Video Encoders */
{
"H.264 Video Encoder", /* String name of codec used by plugin */
"h264enc", /* String name of codec used by CE */
encodeEngine /* Engine that contains this codec */
}, {
"MPEG4 Video Encoder", /* String name of codec used by plugin */
"mpeg4enc", /* String name of codec used by CE */
encodeEngine /* Engine that contains this codec */
},

/* Image Encoders */
{
"JPEG Image Encoder", /* String name of codec used by plugin */
"jpegenc", /* String name of codec used by CE */
encodeEngine /* Engine that contains this codec */
},


No DSP-accelerated Ogg, sadly, but JPEG seems interesting... would it make sense to use it in non-streaming applications (image viewers, web browsers etc)?

[lardman]

It's very different I'm afraid, it supports 8-bit types for example.

[lardman]

We could do JPEG acceleration on the current DSP. I setup a Garage project and started work on this, but other things have got in the way. If anyone's interested then I can have a chat to you about it.

[attila77]

I'd be interested, especially if we could integrate it into pyqtoreader, but my ARM/ASM skills are worse than puny. :(

[lardman]

No need for ASM in the fist instance, the DSP can be programmed in plain C (and use the wrapped macros to access the accelerated functions, etc.). But there are no 8-bit types on the DSP, only a small problem ;)

I was thinking of writing something which has the same sorts of entry points as libjpeg does, so that would certainly be doable.

The only issue is writing the algorithmic stuff to support all the jpeg types, and I've just not had the time to do this. Yet. I'm sure with some support/interest I'll get it sorted.

[qole]

Clearly the next tablet is going to be heavily orientated around its media capabilities. At OSiM last year there was a lot of talk about the camera and what you could do with it, video, photo's etc. I can't see why streaming wouldn't be a big factor too. With the online OVI presence Nokia has, uploading videos and even live streaming would fit in with their slogan "Share photos and videos. Anytime. Anywhere."

I've been thinking more about this; with fast enough hardware (or good hardware encoding), and a good enough camera, you could easily have high-quality, live-to-broadcast streaming video, like Edison Carter in Max Headroom, except your camera is a tiny handheld device.

[sjgadsby]

...with fast enough hardware (or good hardware encoding), and a good enough camera, you could easily have high-quality, live-to-broadcast streaming video...

It's going to depend upon how you define "good enough camera" and "broadcast". A significant percentage of those in broadcasting were aghast when DV footage was allowed on air, and there are still those who refuse to accept HDV as "true" HD. A Nokia handheld is going to be a tough sell in the traditional broadcast or video production markets.

On the other hand, there are others who specifically go for the "YouTube look" when creating web video, and a pocketable, all-in-one device such as that you're envisioning would play well for run-and-gun there. Bill it as a Flip MinoHD with Eye-Fi-like upload and built-in, live streaming.

As a side note, this reminds me the demo GY-DV300U we had on loan from JVC some six or seven years ago. It was too little for too much, but time favorably shifts the price point of technology, and I'd say the idea is coming around again.

[qole]

I just have no idea of the A/V encoding power of the 35xx chipset, nor just how much bandwidth would be required to stream this, but I think a minimum requirement for what I'm talking about would be 640x480, 25 fps, no obvious compression artifacts. I'm sure the "traditional broadcast or video production markets" would be aghast at those specs, but I really believe those numbers are sufficient for live reporter-on-the-spot type broadcasts.

There are other factors, of course. For example, image stabilization has been a problem on recent Nokia devices. A nice clear video stream is nothing if it looks like the camera is being held by a palsied hamster.

[attila77]

From TPS65950 and OMAP3530 Hardware Connectivity User's Guide ( http://focus.ti.com.cn/cn/lit/ug/swcu056a/swcu056a.pdf )

- Image/video/audio (IVA) 2+ accelerator, enabling multistandard (MPEG-4, WMV9, RealVideo®, H263, H264) encoding/decoding at D1 (720 x 480 pixels) 30 frames per second (fps)

So, at least in theory, this should be possible IF the rest of the system can sustain the datarate through the whole camera-gstreamer-encoder-streamserver chain.

[qole]

Bah, I can't get flumotion to start on either my Ubuntu desktop machine, or in my tablet's Debian chroot. I'm not sure what's wrong in each case, but it seems that it might have something to do with the SSL certificate I create during the install...

I'm getting more and more excited as I read about this Flumotion server (http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/)... Looks pretty straightforward to set up a web cam to stream out to the web (http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/help-configuration-assistant-producer-video-webcam.html)...

Makes me wonder if this can be done already using the tablet as the "Video Worker (http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/section-architecture-managers-and-workers.html)" and a desktop computer as the "Manager," "Muxer Worker," and "Streamer Worker". That might take some of the strain off of the poor little tablet.

http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/figures/diagram-architecture-with-workers.png

[sjgadsby]

...I really believe those numbers are sufficient for live reporter-on-the-spot type broadcasts.

And a wise man you are: New iPhone Good Enough For The Evening News (http://cbs4.com/local/iphone.Apple.Gio.2.1054634.html).