[Fwd: Re: [libavg-devel] Audio Support Patch]

[Ulrich von Zadow]

-------- Original Message --------
Subject: 	Re: [libavg-devel] Audio Support Patch
Date: 	Tue, 11 Dec 2007 18:15:41 -0800
From: 	Nick Hebner <hebnern at gmail.com>
To: 	Ulrich von Zadow <uzadow at libavg.de>
References:
<7122baa90712111513s30065c8bjc258f434a5b8c8a7 at mail.gmail.com>
<475F3A0E.9010807 at libavg.de>

On Dec 11, 2007 5:31 PM, Ulrich von Zadow <uzadow at libavg.de
<mailto:uzadow at libavg.de>> wrote:

    Hi Nick,

    incredible! It's high time someone tackled that. Lots of comments
    are inline, and I'll have a longer look this weekend.

    Nick Hebner wrote:
    > Depending on your version, you may need to upgrade to get this
    function.

    Then we need to at least run as before (i.e. without audio) if the
    function isn't there before this can be put in a release - or else
    we'll probably break libavg for lots of people.

    > I have gotten the basics of it working: support for multiple
    > videos playing audio,

    Very nice.

    > per-video volumes controllable from python.

    You're not fading the volume internally, right? Then we'll have
    audible clicks whenever there's a large volume jump.

Currently, I am simply using an SDL function (SDL_MixAudio) to do mixing
of multiple buffers. The documentation of this says that it does
'addition, volume adjustment, and overflow clipping' but it is fairly
basic, and I realize that it will probably have to be replaced. I don't
know much about this topic, do you know of any good resources for
learning more about volume fading?


    > I have not
    > added any formal testing yet (any ideas on how to accomplish
    > this?)

    Jup - lots :-).

    - Add a test to src/video/testvideo that decodes a video with a
    silent audio track. Check if the data delivered is silence (or
    nearly so).
    - Same, but have a sine wave (440 Hz or something) as audio track.
    - Repeat with different sampling rates (once that's supported).
    - Repeat with and without a video channel (once that's supported).
    - Repeat with different video container formats and codecs.
    - Once we add volume fading, we can change the volume during a
    decode and have the test look for changes in the sample value that
    are too large - those are the clicks.

    - Add tests to src/test that runs a video/two videos with audio.

Good ideas, this would have to be accomplished in C++ based test
programs then right?

    Also some hints:
    - You probably know this, but: the test files should be pretty
    short, otherwise the download times for releases and the time the
    tests take go through the roof.
    - The tests should be silent by default, or you'll have to work in a
    soundproof room. (That's also a good reason for adding a separate
    global volume value :-) ).

Yeah, I thought of that too, shouldn't be too bad.

    > There are a number of issues outstandi)ng:
    > - Audio and video streams are not sync'd. They seem to look ok if
    > you use the video stream fps, but will probably get out of sync
    > eventually.
    > - Need to create fancier resampling function to handle mixing of
    > audio with different bitrates/channels.

    ffmpeg can handle audio resampling with just a few lines of code
    :-).

Cool. It seems that SDL can too. Any idea which is better?

    >- Video/FFMpegDecoder should handle audio-only files in addition
    >to a/v or v files
    >- Does this mean that the 'video' node should become a more
    >general 'media' node or something like that?

    Hm. Not sure. A media node becomes confusing (what's the width of an
    audio-only node?).

Good point.

    Maybe add an <audio> node? Or leave audio-only nodes
    (because avg files are layout-oriented) and instead add an audio
    object that can be created, started etc. separate from the avg node
    hierarchy?

    As I said, I'm not sure. Any ideas?

Hmm. An <audio> node would be nice for things like background music
(where the sound simply plays without anything interacting with it).
Also, it might nice to be able to define audio resources in the avg file
instead of having to write python to create them (the avg file in
addition to being layout-oriented, is resource-oriented to some degree
as well). Also, it might be nice to include sounds in the node heirarchy
so that they are associated with other objects in the heirarchy ( i.e. a
sound plays while its containing DivNode is active).

On the other hand, dynamic audio objects might be better suited to quick
sound effects. These could still be accomplished by adding, removing,
and interacting with <audio> nodes in the heirarchy as well, so maybe
this could be filed under the 'better support for dynamic node creation'
endeavor?


    > - Output bitrate/channels should be configurable from an avg file.
    > - Formal testing

    Cheers,

     Uli

-- 

Ulrich von Zadow | +49-172-7872715
Jabber: cocacoder at jabber.berlin.ccc.de
Skype: uzadow