 |
The Open Master Hearing Aid (openMHA)
openMHA
Open community platform for hearing aid algorithm research
|
|
The Open Master Hearing Aid (openMHA)
openMHA
Open community platform for hearing aid algorithm research
|
Plugins are C++ code that is compiled and linked against the openMHA library. The compiler needs be instructed on how to find the openMHA headers and library and to link against the openMHA library. There are two possible options: One can compile openMHA and then create a copy of an example plugin directory and customize from there. See COMPILATION.md for more information on how to compile openMHA.
On Ubuntu is is also possible to install the libopenmha-dev package and include config.mk into the user's Makefile. Example 21 provides an example plugin and Makefile for this scenario.
On the personal hearing lab (PHL) running mahalia, the required packages for compiling openMHA plugins are already installed since version 4.17.0-r1. Example 21 can also be used here.
openMHA contains a small number of example plugins as C++ source code. They are meant to help developers in understanding the concepts of openMHA plugin programming starting from the simplest example and increasing in complexity. This tutorial explains the basic parts of the example files.
The example plugin file example1.cpp demonstrates the easiest way to implement an openMHA Plugin. It attenuates the sound signal in the first channel by multiplying the sound samples with a factor. The plugin class MHAPlugin::plugin_t exports several methods, but only two of them need a non-empty implementation:
prepare() prepare() method in the parent class mha_plugin_t<> is a pure virtual method and needs an implementation so that the plugin class can be instantiated.process() process() method is called whenever a new block of audio arrives and needs signal processing by this plugin.Every plugin implementation should include the 'mha_plugin.hh' header file. C++ helper classes for plugin development are declared in this header file, and most header files needed for plugin development are included by mha_plugin.hh.
The class example1_t inherits from the class MHAPlugin::plugin_t, which in turn inherits from MHAParser::parser_t – the configuration language interface in the method "parse". Our plugin class therefore inherits the "parse" method from MHAParser::parser_t, which integrates the plugin into the global openMHA configuration tree.
The constructor has to accept two parameters of types algo_comm_t and std::string, respectively. In this simple example, we do not make use of them.
The release() method is used to free resources after signal processing. In this simple example, we do not allocate resources, so there is no need to free them.
| signal_info | Contains information about the input signal's dimensions, see mhaconfig_t. |
The prepare() method of the plugin is called before the signal processing starts, when the input signal dimensions like domain, number of channels, frames per block, and sampling rate are known. The prepare() method can check these values and raise an exception if the plugin cannot cope with them, as is done here. The plugin can also change these values if the signal processing performed in the plugin results in an output signal with different parameters. This plugin does not change the signal's parameters, therefore they are not modified here.
| signal | Pointer to the input signal structure mha_wave_t. |
The plugin works with time domain input signal (indicated by the data type mha_wave_t of the process method's parameter). It scales the first channel by a factor of 0.1. The output signal reuses the structure that previously contained the input signal (in-place processing).
Plugins have to export C functions as their interface (to avoid C++ name-mangling issues and other incompatibilities when mixing plugins compiled with different C++ compilers).
This macro takes care of accessing the C++ class from the C functions required as the plugin's interface. It implements the C funtions and calls the corresponding C++ instance methods. Plugin classes should be derived from the template class MHAPlugin::plugin_t to be compatible with the C interface wrapper.
This macro also catches C++ exceptions of type MHA_Error, when raised in the methods of the plugin class, and reports the error using an error flag as the return value of the underlying C function. It is therefore important to note that only C++ exceptions of type MHA_Error may be raised by your plugin. If your code uses different Exception classes, you will have to catch them yourself before control leaves your plugin class, and maybe report the error by throwing an instance of MHA_Error. This is important, because: (1) C++ exceptions cannot cross the plugin interface, which is in C, and (2) there is no error handling code for your exception classes in the openMHA framework anyways.
This is another simple example of openMHA plugin written in C++. This plugin also scales one channel of the input signal, working in the time domain. The scale factor and which channel to scale (index number) are made accessible to the configuration language.
The algorithm is again implemented as a C++ class.
| scale_ch | – the channel number to be scaled |
| factor | – the scale factor of the scaling. |
This class again inherits from the template class MHAPlugin::plugin_t for intergration with the openMHA configuration language. The two data members serve as externally visible configuration variables. All methods of this class have a non-empty implementation.
The constructor invokes the superclass constructor with a string parameter. This string parameter serves as the help text that describes the functionality of the plugin. The constructor registers configuration variables with the openMHA configuration tree and sets their default values and permitted ranges. The minimum permitted value for both variables is zero, and there is no maximum limit (apart from the limitations of the underlying C data type). The configuration variables have to be registered with the parser node instance using the MHAParser::parser_t::insert_item method.
| signal_info | – contains information about the input signal's parameters, see mhaconfig_t. |
The user may have changed the configuration variables before preparing the openMHA plugin. A consequence of this is that it is not sufficient any more to check if the input signal has at least 1 audio channel.
Instead, this prepare method checks that the input signal has enough channels so that the current value of scale_ch.data is a valid channel index, i.e. 0 
scale_ch.data < signal_info.channels. The prepare method does not have to check that 0 scale_ch.data, since this is guaranteed by the valid range setting of the configuration variable.
The prepare method then modifies the valid range of the scale_ch variable, it modifies the upper bound so that the user cannot set the variable to a channel index higher than the available channels. Setting the range is done using a string parameter. The prepare method contatenates a string of the form "[0,n[". n is the number of channels in the input signal, and is used here as an exclusive upper boundary. To convert the number of channels into a string, a helper function for string conversion from the openMHA Toolbox is used. This function is overloaded and works for several data types.
It is safe to assume that the value of configuration variables does not change while the prepare method executes, since openMHA preparation is triggered from a configuration language command, and the openMHA configuration language parser is busy and cannot accept other commands until all openMHA plugins are prepared (or one of them stops the process by raising an exception). As we will see later in this tutorial, the same assumption cannot be made for the process method.
The release method should undo the state changes that were performed by the prepare method. In this example, the prepare method has reduced the valid range of the scale_ch, so that only valid channels could be selected during signal processing.
The release method reverts this change by setting the valid range back to its original value, "[0,[".
The processing function uses the current values of the configuration variables to scale every frame in the selected audio channel.
Note that the value of each configuration variable can change while the processing method executes, since the process method usually executes in a different thread than the configuration interface.
For this simple plugin, this is not a problem, but for more advanced plugins, it has to be taken into consideration. The next section takes a closer look at the problem.
Assume that one thread reads the value stored in a variable while another thread writes a new value to that variable concurrently. In this case, you may have a consistency problem. You would perhaps expect that the value retrieved from the variable either (a) the old value, or (b) the new value, but not (c) something else. Yet generally case (c) is a possibility.
Fortunately, for some data types on PC systems, case (c) cannot happen. These are 32bit wide data types with a 4-byte alignment. Therefore, the values in MHAParser::int_t and MHAParser::float_t are always consistent, but this is not the case for vectors, strings, or complex values. With these, you can get a mixture of the bit patterns of old and new values, or you can even cause a memory access violation in case a vector or string grows and has to be reallocated to a different memory address.
There is also a consistency problem if you take the combination of two "safe" datatypes. The openMHA provides a mechanism that can cope with these types of problems. This thread-safe runtime configuration update mechanism is introduced in example 5.
This example introduces the openMHA Event mechanism. Plugins that provide configuration variable can receive a callback from the parser base class when a configuration variable is accessed through the configuration language interface.
The third example performes the same processing as before, but now only even channel indices are permitted when selecting the audio channel to scale. This restriction cannot be ensured by setting the range of the channel index configuration variable. Instead, the event mechanism of openMHA configuration variables is used. Configuration variables emit 4 different events, and your plugin can connect callback methods that are called when the events are triggered. These events are:
writeaccess
valuechanged
readaccess
prereadaccess
All of these callbacks are executed in the configuration thread. Therefore, the callback implementation does not have to be realtime-safe. No other updates of configuration language variables through the configuration language can happen in parallel, but your processing method can execute in parallel and may change values.
This plugin exposes another configuration variable, "prepared", that keeps track of the prepared state of the plugin. This is a read-only (monitor) integer variable, i.e. its value can only be changed by your plugin's C++ code. When using the configuration language interface, the value of this variable can only be read, but not changed.
The patchbay member is an instance of a connector class that connects event sources with callbacks.
This plugin exposes 4 callback methods that are triggered by events. Multiple events (from the same or different configuration variables) can be connected to the same callback method, if desired.
This example plugin uses the valuechanged event to check that the scale_ch configuration variable is only set to valid values.
The other callbacks only cause log messages to stdout, but the comments in the logging callbacks give a hint when listening on the events would be useful.
The constructor of a monitor variable does not require a parameter for setting the initial value. The only parameter here is the help text describing the contents of the read-only variable. If the initial value should differ from 0, then the .data member of the configuration variable has to be set to the initial value in the plugin constructor's body explicitly, as is done here for demonstration although the initial value of this monitor variable is 0.
Events and callback methods are then connected using the patchbay member variable.
The prepare method checks wether the current setting of the scale_ch variable is possible with the input signal dimension. It does not adjust the range of the variable, since the range alone is not sufficient to ensure all future settings are also valid: The scale channel index has to be even.
The release method is needed for tracking the prepared state only in this example.
The signal processing member function is the same as in example 2.
When the writeaccess or valuechanged callbacks throw an MHAError exception, then the change made to the value of the configuration variable is reverted.
If multiple event sources are connected to a single callback method, then it is not possible to determine which event has caused the callback to execute. Often, this information is not crucial, i.e. when the answer to a change of any variable in a set of variables is the same, e.g. the recomputation of a new runtime configuration that takes all variables of this set as input.
This plugin is the same as example 3 except that it works on the spectral domain (STFT).
The prepare method now checks that the signal domain is MHA_SPECTRUM.
The signal processing member function works on the spectral signal instead of the wave signal as before.
The mha_spec_t instance stores the complex (mha_complex_t) spectral signal for positive frequences only (since the waveform signal is always real). The num_frames member of mha_spec_t actually denotes the number of STFT bins.
Please note that different from mha_wave_t, a multichannel signal in mha_spec_t is stored non-interleaved in the signal buffer.
Some arithmetic operations are defined on struct mha_complex_t to facilitate efficient complex computations. The *= operator used here (defined for real and for complex arguments) is one of them.
When connecting a class that performs spectral processing with the C interface, use spec instead of wave as the domain indicator.
Many algorithms use complex operations to transform the user space variables into run time configurations. If this takes a noticeable time (e.g. more than 100-500 
sec), the update of the runtime configuration can not take place in the real time processing thread. Furthermore, the parallel access to complex structures may cause unpredictable results if variables are read while only parts of them are written to memory (cf. section Consistency). To handle these situations, a special C++ template class MHAPlugin::plugin_t was designed. This class helps keeping all access to the configuration language variables in the configuration thread rather than in the processing thread.
The runtime configuration class example5_t is the parameter of the template class MHAPlugin::plugin_t. Its constructor converts the user variables into a runtime configuration. Because the constructor executes in the configuration thread, there is no harm if the constructor takes a long time. All other member functions and data members of the runtime configurations are accessed only from the signal processing thread (real-time thread).
The plugin interface class inherits from the plugin template class MHAPlugin::plugin_t, parameterised by the runtime configuration. Configuration changes (write access to the variables) will emit a write access event of the changed variables. These events can be connected to member functions of the interface class by the help of a MHAEvents::patchbay_t instance.
The constructor of the runtime configuration analyses and validates the user variables. If the configuration is invalid, an exception of type MHA_Error is thrown. This will cause the openMHA configuration language command which caused the change to fail: The modified configuration language variable is then reset to its original value, and the error message will contain the message string of the MHA_Error exception.
In this example, the run time configuration class example5_t has a signal processing member function. In this function, the selected channel is scaled by the given scaling factor.
The constructor of the example plugin class is similar to the previous examples. A callback triggered on write access to the variables is registered using the MHAEvents::patchbay_t instance.
The processing function can gather the latest valid runtime configuration by a call of poll_config. On success, the class member cfg points to this configuration. On error, if there is no usable runtime configuration instance, an exception is thrown. In this example, the prepare method ensures that there is a valid runtime configuration, so that in this example, no error can be raised at this point. The prepare method is always executed before the process method is called. The runtime configuration class in this example provides a signal processing method. The process method of the plugin interface calls the process method of this instance to perform the actual signal processing.
The prepare method ensures that a valid runtime configuration exists by creating a new runtime configuration from the current configuration language variables. If the configuraion is invalid, then an exception of type MHA_Error is raised and the preparation of the openMHA fails with an error message.
The update_cfg member function is called when the value of a configuration language variable changes, or from the prepare method. It allocates a new runtime configuration and registers it for later access from the real time processing thread. The function push_config stores the configuration in a FiFo queue of runtime configurations. Once they are inserted in the FiFo, the MHAPlugin::plugin_t template is responsible for deleting runtime configuration instances stored in the FiFo. You don't need to keep track of the created instances, and you must not delete them yourself.
In the end of the example code file, the macro MHAPLUGIN_CALLBACKS defines all ANSI-C interface functions and passes them to the corresponding C++ class member functions (partly defined by the MHAPlugin::plugin_t template class). All exceptions of type MHA_Error are caught and transformed into an appropriate error code and error message.
This example is the same as the previous one, except that it additionally creates an 'Algorithm Communication Variable' (AC variable). It calculates the RMS level of a given channel and stores it into this variable. The variable can be accessed by any other algorithm in the same chain. To store the data onto disk, the 'acsave' plugin can be used. 'acmon' is a plugin which converts AC variables into parsable monitor variables.
In the constructor of the plugin class the variable rmsdb is registered under the name example6_rmslev as a one-dimensional AC variable of type float. For registration of other types, read access and other detailed informations please see Communication between algorithms.
Suppose you would want to step through the code of your openMHA plugin with a debugger. This example details how to use the GDB debugger to inspect the example6_t::prepare() and example6_t::process() routines of example6.cpp example 6.
First, make sure that your plugin is compiled with the compiler option to include debugging symbols: Apply the -ggdb switch to all gcc, g++ invocations.
Once the plugin is compiled with debugging symbols, create a test configuration. For example 6, assuming there is an audio file named input.wav in your working directory, you could create a configuration file named ‘debugexample6.cfg’, with the following content:
# debugexample6.cfg fragsize = 64 srate = 44100 nchannels_in = 2 iolib = MHAIOFile io.in = input.wav io.out = output.wav mhalib = example6 mha.channel = 1 cmd=start
Assuming all your binaries and shared-object libraries are in your ‘bin’ directory (see README.md), you could start gdb using
$ export MHA_LIBRARY_PATH=$PWD/bin $ gdb $MHA_LIBRARY_PATH/mha
Set breakpoints in prepare and process methods, and start execution. Note that specifying the breakpoint by symbol (example6_t::prepare) does not yet work, as the symbol lives in the openMHA plugin that has not yet been loaded. Specifying by line number works, however. Specifying the breakpoint by symbol also works once the plugin is loaded (i.e. when the debugger stops in the first break point). You can set the breakpoints like this (example shown here is run in gdb version 7.11.1):
(gdb) run ?read:debugexample6.cfg
Starting program: {openMHA_directory}/bin/mha ?read:debugexample6.cfg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
The Open Master Hearing Aid (openMHA) server
Copyright (c) 2005-2021 HoerTech gGmbH, D-26129 Oldenburg, Germany
This program comes with ABSOLUTELY NO WARRANTY; for details see file COPYING.
This is free software, and you are welcome to redistribute it
under the terms of the GNU AFFERO GENERAL PUBLIC LICENSE, Version 3;
for details see file COPYING.
Breakpoint 1, example6_t::prepare (this=0x6478b0, tfcfg=...)
at example6.cpp:192
192 if( tfcfg.domain != MHA_WAVEFORM )
(gdb) b example6.cpp:162
Breakpoint 2 at 0x7ffff589744a: file example6.cpp, line 162.
(gdb) c
Continuing.
Where ‘{openMHA_directory}’ is the directory where openMHA is located (which should also be your working directory in this case). Next stop is the process() method. You can now examine and change the variables, step through the program as needed (using, for example ‘n’ to step in the next line):
Breakpoint 2, example6_t::process (this=0x7ffff6a06c0d, wave=0x10a8b550)
at example6.cpp:162
162 {
(gdb) n
163 poll_config();
(gdb)
This section introduces how to test a plugin with C++ unit tests using the Googletest framework. In order to execute the tests, navigate to the openMHA root directory and run make unit-tests in your terminal. Afterwards you may execute make unit-tests in the plugin directory in order to only execute the very test you are working on.
As an example, unit tests for plugin example7.cpp are written, which is functionally the same as plugin example1.cpp (see section example1.cpp). In order to write unit tests for your plugin it must have its class/function declarations in a header file (.hh) so you can include it in the unit test file. The class/function definitions are contained in the respective source file (.cpp).
The unit tests are written using a test fixture class (here: example7_testing) which will be inherited by the individual tests (TEST_F). This enables us to use the members in example7_testing in multiple tests without the need for redundant declarations.
The test fixture class is derived from the ::testing::Test class declared in gtest.h. The constructor of example7_t needs three parameters, namely a handle to the algorithm communication variable space and two strings. A container for audio signals for repeatedly passing blocks of the input signal to the plugin under test is also allocated by the test fixture class. It is defined as an instance of MHASignal::waveform_t with the name wave_input and its values are zero upon initialization.
The first test checks whether the state methods work as expected. Next to the actual processing there are often certain variables in each individual openMHA plugin that need to be allocated beforehand or wiped from memory afterwards. The methods that are used to do this are prepare() and release(). In order to assert that they were called and that we switched states accordingly we use the methods prepare_() and release_() (Note: the underscore!) that are defined in the plugin base class mha_plugin_t<>. These methods keep track of the state, call prepare() and release() and do additional bookkeeping. To ensure that the state methods work as expected the Googletest methods EXPECT_FALSE and EXPECT_TRUE are used.
In this test the goal is to assess the main feature of the plugin (example7_t), which is the same as in example1_t, namely altering the signal's first channel by a constant factor of 0.1. The variable wave_input is the signal that will be processed by the plugin. In order to assert the success, the elements in wave_input are set to a constant value of 1, because they are 0 upon initialization. During the process() function all elements of the first channel of wave_input are multiplied by the factor 0.1. Before process() is called the value assigned to wave_input is checked via the method EXPECT_FLOAT_EQ provided by Googletest. The values of wave_input are retrieved by the method value() by passing the desired sample position and channel number as second and third input parameter, respectively. Here, we checked the values of two frames in each channel to show the difference before and after processing; the frame indices were chosen randomly. After calling process(), the values contained in wave_input are checked again to make sure that the plugin worked as intended.
