Downloaded 559 times
Uploaded byAlice Zheng
Understanding Feature Space in Machine Learning
The document discusses the importance of feature engineering in machine learning, highlighting how raw data is transformed into high-dimensional vectors for modeling and predictions. It outlines various methods for representing data, such as bag-of-words and term frequency-inverse document frequency (tf-idf), and emphasizes the challenges of visualizing and understanding high-dimensional feature spaces. Additionally, it touches on the broader implications of geometric versus algebraic approaches in understanding machine learning models.
In this document
Powered by AI
Overview of Machine Learning and its context from Alice Zheng, emphasizing feature space.
Discussion on Alice's journey in applied machine learning, the necessity for experts, and tool development.
Exploration of key reasons for utilizing machine learning: data modeling, predictions, and intelligent applications.
Overview of machine learning pipeline stages, from raw data to model deployment.
Definition of features as numeric representations of raw data.
Methods for representing natural text using Bag of Words for classification (e.g., puppy classification).
Explaining raw and visual representations of images, using bag of visual words and dense vector representation.
Description of feature space in machine learning, including point clouds and feature engineering.
Discussion on algebra vs. geometry in mathematics and their applications in understanding machine learning.
Graphical representation of spheres in different dimensions (2D, 3D, 4D) and its significance.
How higher dimensions help in modeling complex processes like the birth and death processes.
Introductory exploration of visualizing feature space in machine learning.
The complications and limitations encountered in visualizing and managing high-dimensional feature space.
Clarifying bag-of-words representation through visual examples and data representation.
Discussion on document point clouds and their role in machine learning.
Definition of model as a mathematical summary of data using geometric shapes.
Illustration of different models in machine learning: classification, clustering, and regression.
Concept of feature engineering and its importance in machine learning illustrated through visuals.
Identifying limitations of the bag-of-words model in text classification tasks.
Introduction of tf-idf as an improvement over bag-of-words for text representation.
Discussion on how tf-idf adjusts the decision surface for features in the context of classification.
Initial steps in feature engineering are discussed, focusing on data representation and modeling methods.
Summary of feature engineering concepts in machine learning, with a recruitment note for Dato.
More Related Content
Similar to Understanding Feature Space in Machine Learning
Understanding Feature Space in Machine Learning
- 1. Understanding Feature Space in MachineLearning Alice Zheng, Dato September 9, 2015 1
- 2. 2 My journey sofar Applied machine learning (Data science) Build ML tools Shortage of experts and good tools.
- 3. 3 Why machine learning? Modeldata. Make predictions. Build intelligent applications.
- 4. 4 The machine learningpipeline I fell in love the instant I laid my eyes on that puppy. His big eyes and playful tail, his soft furry paws, … Raw data Features Models Predictions Deploy in production
- 5. Feature = numericrepresentation of raw data
- 6. 6 Representing natural text Itis a puppy and it is extremely cute. What’s important? Phrases? Specific words? Ordering? Subject, object, verb? Classify: puppy or not? Raw Text {“it”:2, “is”:2, “a”:1, “puppy”:1, “and”:1, “extremely”:1, “cute”:1 } Bag of Words
- 7. 7 Representing natural text Itis a puppy and it is extremely cute. Classify: puppy or not? Raw Text Bag of Words it 2 they 0 I 1 am 0 how 0 puppy 1 and 1 cat 0 aardvark 0 cute 1 extremely 1 … … Sparse vector representation
- 8. 8 Representing images Image source:“Recognizing and learning object categories,” Li Fei-Fei, Rob Fergus, Anthony Torralba, ICCV 2005—2009. Raw image: millions of RGB triplets, one for each pixel Classify: person or animal? Raw Image Bag of Visual Words
- 9. 9 Representing images Classify: person oranimal? Raw Image Deep learning features 3.29 -15 -5.24 48.3 1.36 47.1 - 1.92 36.5 2.83 95.4 -19 -89 5.09 37.8 Dense vector representation
- 10. 10 Feature space inmachine learning • Raw data high dimensional vectors • Collection of data points point cloud in feature space • Model = geometric summary of point cloud • Feature engineering = creating features of the appropriate granularity for the task
- 11. Crudely speaking, mathematiciansfall into two categories: the algebraists, who find it easiest to reduce all problems to sets of numbers and variables, and the geometers, who understand the world through shapes. -- Masha Gessen, “Perfect Rigor”
- 12. 12 Algebra vs. Geometry a b c a2+ b2 = c2 Algebra Geometry Pythagorean Theorem (Euclidean space)
- 13. 13 Visualizing a spherein 2D x2 + y2 = 1 a b c Pythagorean theorem: a2 + b2 = c2 x y 1 1
- 14. 14 Visualizing a spherein 3D x2 + y2 + z2 = 1 x y z 1 1 1
- 15. 15 Visualizing a spherein 4D x2 + y2 + z2 + t2 = 1 x y z 1 1 1
- 16. 16 Why are welooking at spheres? = = = = Poincaré Conjecture: All physical objects without holes is “equivalent” to a sphere.
- 17. 17 The power ofhigher dimensions • A sphere in 4D can model the birth and death process of physical objects • Point clouds = approximate geometric shapes • High dimensional features can model many things
- 18.
- 19. 19 The challenge ofhigh dimension geometry • Feature space can have hundreds to millions of dimensions • In high dimensions, our geometric imagination is limited - Algebra comes to our aid
- 20. 20 Visualizing bag-of-words puppy cute 1 1 I havea puppy and it is extremely cute I have a puppy and it is extremely cute it 1 they 0 I 1 am 0 how 0 puppy 1 and 1 cat 0 aardvark 0 zebra 0 cute 1 extremely 1 … …
- 21. 21 Visualizing bag-of-words puppy cute 1 1 1 extremely I havea puppy and it is extremely cute I have an extremely cute cat I have a cute puppy
- 22. 22 Document point cloud word1 word 2
- 23. 23 What is amodel? • Model = mathematical “summary” of data • What’s a summary? - A geometric shape
- 24. 24 Classification model Feature 2 Feature1 Decide between two classes
- 25. 25 Clustering model Feature 2 Feature1 Group data points tightly
- 26. 26 Regression model Target Feature Fit thetarget values
- 27.
- 28. 28 When does bag-of-wordsfail? puppy cat 2 1 1 have I have a puppy I have a cat I have a kitten Task: find a surface that separates documents about dogs vs. cats Problem: the word “have” adds fluff instead of information I have a dog and I have a pen 1
- 29. 29 Improving on bag-of-words •Idea: “normalize” word counts so that popular words are discounted • Term frequency (tf) = Number of times a terms appears in a document • Inverse document frequency of word (idf) = • N = total number of documents • Tf-idf count = tf x idf
- 30. 30 From BOW totf-idf puppy cat 2 1 1 have I have a puppy I have a cat I have a kitten idf(puppy) = log 4 idf(cat) = log 4 idf(have) = log 1 = 0 I have a dog and I have a pen 1
- 31. 31 From BOW totf-idf puppy cat1 have tfidf(puppy) = log 4 tfidf(cat) = log 4 tfidf(have) = 0 I have a dog and I have a pen, I have a kitten 1 log 4 log 4 I have a cat I have a puppy Decision surface Tf-idf flattens uninformative dimensions in the BOW point cloud
- 32. 32 Entry points offeature engineering • Start from data and task - What’s the best text representation for classification? • Start from modeling method - What kind of features does k-means assume? - What does linear regression assume about the data?
- 33. 33 That’s not all,folks! • There’s a lot more to feature engineering: - Feature normalization - Feature transformations - “Regularizing” models - Learning the right features • Dato is hiring! jobs@dato.com alicez@dato.com @RainyData
Editor's Notes
- #5 Features sit between raw data and model. They can make or break an application.