Advanced Data Science Techniques

The predictive efficacy of the machine-learning algorithms is greatly amplified by feature engineering from the raw input feature-space. A meticulous process of exploratory data analysis (EDA) in search of clues towards meaningful feature extraction often can transform a simple algorithm which was performing very poorly on the original feature-space into an extraordinarily effective one in the new feature space where these extracted features are present.

Approximate & Kernel k-NN is a lazy-learning, non-parametric method that proves remarkably effective in certain situations. We will learn about the interesting notions of distances or similarities that underly the search for nearest neighbors. We will learn about the Curse of Dimensionality, its origin, and the methods to deal with it. We will study how the choice of the “k” provides the bias-variance tradeoffs. Finally, we will learn about a rich collection of distance kernels that mitigate the need for hyperparameter “k” tuning.

Decision Tree provides a versatile means to adapt to non-linearities in the feature space. It is employed for classification as well as regression. The strength of decision trees resides in their very intuitive representation as a tree structure in the way it iteratively partitions the feature space. Thus it is valued for its interpretability, which at the same time being remarkably powerful.

Among the most powerful concepts to have emerged in machine learning is that of ensemble learning. Instead of using only one learner, and making it as powerful as possible so that it comes up with a good hypothesis, the approach here is different. Instead, a crowd of weak learners is taken, each forming its own hypothesis. These hypotheses are methodically synthesized to generate predictions in a manner that there is much greater overall predictive power.

RandomForest is a powerful approach to both classification and regression that employs an ensemble of decision trees and bagging to get a consensus vote or score. Extremely Randomized Trees are a variation of the same approach but exhibit fewer variance errors.

Gradient Boosting is a powerful ensemble method that has proven remarkably effective in recent years. As a result, there is a lot of research and activity in creating continually better performing and more predictive implementations.

Support Vector Machines provide a systematic way to linear the decision boundary by transforming the original feature space into another where the various classes are separated by a linear maximal margin classifier.

The No Free Lunch theorems — despite their frivolous sounding name — are a fundamental guideline that no one algorithm is more “powerful” than any other, on the average. Different algorithms make different underlying assumptions of the ground truth and therefore are well suited to different datasets. On average, they all perform equivalently.