What are MFCCs and how are they computed?
Feature extraction is the first step in any automatic speech recognition (ASR) pipeline. The objective is to compute features from speech waveforms which contain relevant information about the linguistic content of the speech, and ignore information about the background noise, emotions, etc. This problem has been extensively studied since early days in ASR research, and several feature extraction methods have been proposed. Among these, the most wellknown and widely used are Mel Frequency Cepstral Coefficients (MFCCs).
Since MFCCs are very well known, I will only briefly describe their computation in this post. Most of this is taken from this blog, which explains them in some detail. The key steps for computing MFCCs are described below.

First, the entire waveform is divided into shorter segments of 2040 ms each. The assumption is that in this short segment, the signal is statistically stationary, and so features can be assumed to be constant inside this window. In Kaldi and most major ASR systems, windows are 25 ms in length and at 10 ms intervals apart, i.e., they are overlapping.

In order to recognize the frequencies present in this short segment, the power spectrum (or the periodogram estimate) is computed. This is done using discretetime Fourier transforms.

It is difficult to distinguish individual frequencies in the raw power spectrum, especially in the high frequency range. To solve this problem, the spectrum is convolved with several (2040, in general) triangular Mel filters, called a filterbank. These filters are narrow at low frequency and get wider as frequency increases, in accordance with the human cochlea. Furthermore, a log transform is applied since humans don’t perceive loudness on a linear scale.

Since filterbank energies are correlated and cannot be used directly with a Gaussian mixture with diagonal covariance, we apply a discrete cosine transform (DCT) to decorrelate them.
There is some debate in the community regarding the use of the DCT, instead of directly using the log Mel fiterbank features, particularly for deep neural network based acoustic models. Some research groups, like Google, use filterbanks (fbanks) while Kaldi mostly uses MFCCs, especially in its TDNN chain models. Here is Dan Povey’s take on this:
The reason we use MFCC is because they are more easily compressible, being decorrelated; we dump them to disk with compression to 1 byte per coefficient. But we dump all the coefficients, so it’s equivalent to filterbanks times a fullrank matrix, no information is lost.
(Source: kaldihelp)
Delta and deltadelta features
The idea behind using delta (differential) and deltadelta (acceleration) coefficients is that in order to recognize speech better, we need to understand the dynamics of the power spectrum, i.e., the trajectories of MFCCs over time. The delta coeffients are computed using the following formula.
\(d_t = \frac{\sum_{n=1}^N n (c_{t+n}  c_{tn})}{2 \sum_{n=1}^N n^2},\) where $d_t$ is a delta coefficient from frame $t$ computed in terms of the static coefficients $c_{tn}$ to $c_{t+n}$. $n$ is usually taken to be 2. The acceleration coefficients are computed similarly, but using the differential instead of the static coefficients.
The LDA transform in Kaldi
For a comprehensive reference on LDA, readers are advised to refer to this post.
The latest TDNNbased chain models in Kaldi (see, for example, this recipe) do not use differential and acceleration features (hereby refered to as “delta features” for convenience). Instead, they employ an LDAlike transformation which is essentially an affine transformation of the spliced input. Here is a sample from the xconfig of a typical Kaldi TDNN model:
input dim=100 name=ivector
input dim=40 name=input
# please note that it is important to have input layer with the name=input
# as the layer immediately preceding the fixedaffinelayer to enable
# the use of short notation for the descriptor
fixedaffinelayer name=lda input=Append(1,0,1,ReplaceIndex(ivector, t, 0)) affinetransformfile=$dir/configs/lda.mat
# the first splicing is moved before the lda layer, so no splicing here
relubatchnormdropoutlayer name=tdnn1 $tdnn_opts dim=1024
tdnnflayer name=tdnnf2 $tdnnf_opts dim=1024 bottleneckdim=128 timestride=1
tdnnflayer name=tdnnf3 $tdnnf_opts dim=1024 bottleneckdim=128 timestride=1
This splicing can be over 1 or 2 frames on either side of the central frame, i.e. Append(1,0,1)
or Append(2,1,0,1,2)
. Additionally, ivectors are appended with the spliced input before the LDA. Although Kaldi itself has an implementation of the LDA transform available, the transformation here simply multiplies the spliced input with a fullrank matrix. This is why this is called an “LDAlike”, and not an LDA transform.
Some new results
In some sense, this LDAlike transform is a generalization of using the delta features, since it can apply arbitrary scaling to each coefficient, and this matrix is learned in the training stage. However, this means having to additionally learn $(k \times n+d)^2$ parameters, where $k$ is the splicing window, $n$ is the MFCC size, and $d$ is the ivector dimensionality. For typical values of $k$, $n$, and $d$, this is in the range of 50000 to 90000 parameters. While this is not a “huge” number compared to the size of modern deep networks (a typical TDNN model in Kaldi may have up to 10 million parameters), we would still like to see if this is disposable.
I replaced the LDA transform with simple delta features. In the context of our input, the differential is simply $c_{t+1}  c_{t1}$, and the acceleration is $c_{t2} + c_{t+2}  2\times c_t$. This is implemented using a new xconfig
layer called deltalayer
as follows.
class XconfigDeltaLayer(XconfigLayerBase):
"""This class is for parsing lines like
'deltalayer name=delta input=idct'
which appends the central frame with the delta features
(i.e. 1,0,1 since scale equals 1) and deltadelta features
(i.e. 1,0,2,0,1), and then applies batchnorm to it.
Parameters of the class, and their defaults:
input='[1]' [Descriptor giving the input of the layer]
"""
def __init__(self, first_token, key_to_value, prev_names=None):
XconfigLayerBase.__init__(self, first_token, key_to_value, prev_names)
def set_default_configs(self):
self.config = {'input': '[1]'}
def check_configs(self):
pass
def output_name(self, auxiliary_output=None):
assert auxiliary_output is None
return self.name
def output_dim(self, auxiliary_output=None):
assert auxiliary_output is None
input_dim = self.descriptors['input']['dim']
return (3*input_dim)
def get_full_config(self):
ans = []
config_lines = self._generate_config()
for line in config_lines:
for config_name in ['ref', 'final']:
# we do not support user specified matrices in this layer
# so 'ref' and 'final' configs are the same.
ans.append((config_name, line))
return ans
def _generate_config(self):
# by 'descriptor_final_string' we mean a string that can appear in
# configfiles, i.e. it contains the 'final' names of nodes.
input_desc = self.descriptors['input']['finalstring']
input_dim = self.descriptors['input']['dim']
output_dim = self.output_dim()
configs = []
line = ('dimrangenode name={0}_copy1 inputnode={0} dim={1} dimoffset=0'.format(
input_desc, input_dim))
configs.append(line)
line = ('dimrangenode name={0}_copy2 inputnode={0} dim={1} dimoffset=0'.format(
input_desc, input_dim))
configs.append(line)
line = ('component name={0}_2 type=NoOpComponent dim={1}'.format(
input_desc, output_dim))
configs.append(line)
line = ('componentnode name={0}_2 component={0}_2 input=Append(Offset({0},0),'
' Sum(Offset(Scale(1.0,{0}_copy1),1), Offset({0},1)), Sum(Offset({0},2), Offset({0},2),'
' Offset(Scale(2.0,{0}_copy2),0)))'.format(input_desc))
configs.append(line)
line = ('component name={0} type=BatchNormComponent dim={1}'.format(
self.name, output_dim))
configs.append(line)
line = ('componentnode name={0} component={0} input={1}_2'.format(
self.name, input_desc))
configs.append(line)
return configs
The following are some experimental results on mini_librispeech
, wsj
(Wall Street Journal), and swbd
(Switchboard). The ivector scale was reduced for mini_librispeech
since the delta features are computed on top of a SpecAugment layer, which itself includes batch normalization. Therefore, using an ivector scale of 1.0 would overpower the MFCCs.
Setup  Test set  IDCT  SpecAugment  ivector scale  LDA  Delta 

mini_librispeech  dev_clean2  Y  Y  0.5  7.54  7.66 
wsj  eval92  Y  N  1.0  2.39  2.41 
swbd  rt03  N  N  1.0  15.0  15.0 
These results are for a particular test set for these setups, and for a specific decoder, but the general trend of results is found to be the same across all test set and decoder combinations. Without significant loss in performance, we can eliminate the need of an LDA transform in the network. Work on a pull request for this setup is in progress.