Dissecting GPT-Live
This post was jointly written by Desh and Claude, based on Desh’s notes.
Thoughts on machine learning, speech processing, and research. 61 articles
Notes from the Ultra-scale Playbook
These are my notes from reading the Ultra-scale Playbook by HuggingFace. These are only meant for quick review and summary of concepts.
Are Decoder-only Models the Future of Streaming ASR?
A few days ago, I conducted a poll soliciting opinions about the following question:
Notes from the RLHF book
Book Link
Transducers at InterSpeech 2023
Neural transducers are the most popular ASR modeling paradigm in both academia and industry. Since I could not attend InterSpeech 2023 in person, I decided...
GBO notes: Approximation algorithms
This note is a brief introduction to approximation algorithms. Basically, the “Intro to Algorithms” courses are concerned with problems which are solvable in poly-time (i.e.,...
Heilmeier Catachism of my research
Since I am preparing for my GBO exam (which is a kind of qualifying exam where a committee evaluates your preparedness towards your PhD), my...
GBO notes: Machine learning basics (Part 5)
In this series of notes we will review some basic concepts that are usually covered in an Intro to ML course. These are based on...
GBO notes: Machine learning basics (Part 4)
In this series of notes we will review some basic concepts that are usually covered in an Intro to ML course. These are based on...
GBO notes: Machine learning basics (Part 3)
In this series of notes we will review some basic concepts that are usually covered in an Intro to ML course. These are based on...
GBO notes: Machine learning basics (Part 2)
In this series of notes we will review some basic concepts that are usually covered in an Intro to ML course. These are based on...
GBO notes: Machine learning basics (Part 1)
In this series of notes we will review some basic concepts that are usually covered in an Intro to ML course. These are based on...
GBO notes: Continuous speech separation
In the previous post on the MVDR beamformer, we saw how speaker-specific “masks” can be used in conjunction with a multi-channel input signal to extract...
GBO notes: MVDR beamforming
In a previous note, we described the process of mask estimation using complex angular central GMMs that are used in guided source separation (GSS). Mask...
GBO notes: Mask estimation for GSS
Guided source separation (GSS) is an unsupervised algorithm for target speech extraction, first proposed in the Paderborn submission to the CHiME-5 challenge. Given a noisy...
GBO notes: Variational Bayes and the VBx algorithm
Speaker diarization is often formulated as a clustering of speaker embeddings. If we use conventional clustering methods such as k-means or spectral clustering, they ignore...
GBO notes: Spectral clustering
In this note, I will review a popular clustering algorithm called spectral clustering. We will discuss its connection to the min-cut problem in graph partitioning,...
GBO notes: i-vectors and x-vectors
In this note, we will review the two most popular speaker embedding extraction methods, namely i-vectors and x-vectors. But first, it would be useful to...
GBO notes: Expectation Maximization
In this note, we will describe how to estimate the parameters of GMM and HMM models using expectation-maximization method. The equations and discussion is heavily...
A round-up of linear transformers
Introduction
My 3 takeaways from IEEE ICASSP 2021
I attended the virtual ICASSP 2021, and this is a short post with my 3 key take-aways from the conference. As with my previous conference...
On the evaluation of speaker diarization systems
Recently, I have been involved in several projects related to speaker diarization. For the uninitiated, diarization is defined as the task of partitioning a recording...
Desh's curated list of ASR, diarization, and related papers from Interspeech 2020
Interspeech 2020 just ended, and here is my curated list of papers that I found interesting from the proceedings.
Understanding the JHU CHiME-6 Submission - Part I (Enhancement)
The CHiME-6 challenge concluded last month and our team from JHU was ranked 2nd in Track 2 (“diarization + ASR” track). For a reader unfamiliar...
How to use the pre-trained Librispeech model in Kaldi
[Update on Feb 25, 2022] The pre-trained model did not have a frame_subsampling_factor file, which is required for correct decoding. This has now been added...
The Challenges of using Transformers in ASR
Since mid 2018 and throughout 2019, one of the most important directions of research in speech recognition has been the use of self-attention networks and...
Some interesting papers from ASRU 2019
The IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) 2019 ended last week, and here is a list (with very brief summaries) of some...
Highlights from SANE 2019
I attended the Speech and Audio in the Northeast (SANE) 2019 conference at Columbia University last Thursday, and in this post, I will try to...
A Reading List for Automatic Speech Recognition
Here is a list of papers and book chapters for someone who wants to get started with ASR. This has been compiled from the required...
ACL 2019: Notes from an ASR perspective
I did not attend ACL 2019 in Florence, Italy. I did, however, go through several videos (all the videos of oral presentations are available here),...
A note on MFCCs and delta features
What are MFCCs and how are they computed?
On lattice free MMI and Chain models in Kaldi
Update (January 22, 2020): After several discussions with Matthew Wiesner, I have added some content to this post (e.g. deriving the derivatives for MMI) and...
Some Kaldi Things
This is a regularly updated post on some tips and tricks for working with Kaldi.
Iterative Scaling and Coordinate Descent
Recently, I was reading a paper on language model adaptation, which used an optimization technique called Generalized Iterative Scaling (GIS). Having no idea what the...
Experiments with Subword Modeling
Think about tasks such as machine translation (MT), automatic speech recognition (ASR), or handwriting recognition (HWR). While these appear very distinct, on abstraction they share...
Award-winning classic papers in ML and NLP
I was trying to find a consolidated list of papers in machine learning (ICML, NIPS, AAAI, SIGIR) and natural language processing (ACL, EMNLP, NAACL) published...
Theory of Deep Learning: An Illustration with Embeddings
We have discussed several aspects of deep learning theory, ranging from optimization and generalization guarantees to role of depth and generative models. In this final...
Theory of Deep Learning: Generative Models
Till now, in this series based on the ICML 2018 tutorial on “Toward a Theory for Deep Learning” by Prof. Sanjeev Arora, we have limited...
Theory of Deep Learning: Role of Depth
In the previous posts of this series, we have looked at how stochastic gradient descent is able to find a good solution despite the nonconvex...
Theory of Deep Learning: Generalization
In Part 1 of this series, based on the ICML 2018 tutorial on “Toward a Theory for Deep Learning” by Prof. Sanjeev Arora, we looked...
Theory of Deep Learning: Optimization
I only just got around to watching the ICML 2018 tutorial on “Toward a Theory for Deep Learning” by Prof. Sanjeev Arora. In this and...
Transfer Learning in NLP
Transfer learning is undoubtedly the new (well, relatively anyway) hot thing in deep learning right now. In vision, it has been in practice for some...
Evaluation Methods for Dialog Systems
Spoken Dialog Systems (SDS) have become very popular recently, especially for goal completion tasks on mobile devices. Also, with the increasing use of IoT devices...
The 8 Commandments for Coding Your Research
This article is in a different flavor from the other posts in this publication. This is because I have been reading Roman Vershynin’s “High Dimensional...
An Introduction to Speech Recognition using WFSTs
Until now, all of my blog posts have been about deep learning methods or their application to NLP. Since the last couple of weeks, however,...
How to Obtain Sentence Vectors
In several of my previous posts, I have discussed methods for obtaining word embeddings, such as SVD, word2vec, or GloVe. In this post, I will...
Online Learning of Word Embeddings
Word vectors have become the building blocks for all natural language processing systems. I have earlier written an overview of popular algorithms for learning word...
Sparsity in Online Learning with Lasso Regularization
Sparse vectors have become popular recently for 2 reasons:
Irony Detection in Tweets
There was a SemEval 2018 Shared Task on “irony detection in tweets” that ended recently. As a fun personal project, I thought of giving it...
Introduction to Learning Theory - Part 2
In the first part of this series on learning theory, we looked only at the case of finite hypothesis sets, and derived some generalization bounds...
Introduction to Learning Theory - Part 1
One of the most significant take-aways from NIPS 2017 was the “alchemy” debate spearheaded by Ali Rahimi. In the wake of the event, I have...
Unsupervised Approaches for NMT
Translation is one of those tasks in language where the arrival of deep learning systems, and in particular sequence-to-sequence, has been something like a boon....
Beyond Euclidean Embeddings
Representation learning, as the name suggests, seeks to learn representations for structures such as images, videos, words, sentencences, graphs, etc., which may then be used...
Deep Learning for Multimodal Systems
When I was browsing through research groups for my grad school applications, I came across some interesting applications of new deep learning methods in a...
Trends in Semantic Parsing - Part 2
In Part 1 of this two-part series, I discussed some supervised approaches for the objective. In this part, we will look at some unsupervised or...
The Best Papers at ICLR 2017
The International Conference on Learning Representations (ICLR) has evolved into the deep learning conference over the last few years, and with its open review system,...
The Last 3 Years in Text Classification
While working on my undergrad thesis on relation classification of biomedical text using deep learning methods, I quickly hacked together models in Tensorflow that combined...
Understanding Word Vectors
This article is a formal representation of my understanding of vector semantics, from course notes and reading reference papers and chapters from Jurafsky’s SLP book....
Trends in Semantic Parsing - Part 1
In this article, I will try to round up some (mostly neural) approaches for semantic parsing and semantic role labeling (SRL). This is not an...
Metrics for NLG Evaluation
Simple natural language processing tasks such as sentiment analysis, or even more complex ones like semantic parsing are easy to evaluate since the evaluation simply...