Download as PDF, PPTX
![Using Optuna for CASH Problems
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py](https://image.slidesharecdn.com/automl-190916020403/85/Python-AutoML-at-PyConJP-2019-43-320.jpg)
![Optuna for CASH Problem
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py
Algorithm Selection](https://image.slidesharecdn.com/automl-190916020403/85/Python-AutoML-at-PyConJP-2019-44-320.jpg)
![Optuna for CASH Problem
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py
Hyperparameter optimization](https://image.slidesharecdn.com/automl-190916020403/85/Python-AutoML-at-PyConJP-2019-45-320.jpg)
![Using Optuna for CASH Problems
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py](https://image.slidesharecdn.com/automl-190916020403/75/Python-AutoML-at-PyConJP-2019-43-2048.jpg)
![Optuna for CASH Problem
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py
Algorithm Selection](https://image.slidesharecdn.com/automl-190916020403/75/Python-AutoML-at-PyConJP-2019-44-2048.jpg)
![Optuna for CASH Problem
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py
Hyperparameter optimization](https://image.slidesharecdn.com/automl-190916020403/75/Python-AutoML-at-PyConJP-2019-45-2048.jpg)
Uploaded byMasashi Shibata
PythonとAutoML at PyConJP 2019
The document discusses various aspects of automated machine learning (AutoML) in Python, covering feature preprocessing, model selection, and hyperparameter optimization techniques. It references notable tools and frameworks, such as Auto-Sklearn, TPOT, Hyperopt, and Optuna, while highlighting methods for feature selection and data cleaning. Additionally, it presents insights from multiple research papers and tutorials related to the advancement of AutoML methodologies.
More Related Content
Similar to PythonとAutoML at PyConJP 2019
![RTP_AR_Basic_Learners' Workbook_KS2 [FOR REPRODUCTION] (1).pdf](https://cdn.slidesharecdn.com/ss_thumbnails/rtparbasiclearnersworkbookks2forreproduction1-251016024943-e51a16ac-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
PythonとAutoML at PyConJP 2019
- 1. Feature Preprocessing Raw Data Feature Selection Feature Model Selection Data Cleaning AutomatedMachine Learning in Python PyCon JP 2019 AI Lab Python AutoML Feature Preprocessing Raw Data Feature Selection Feature Model Selection Data Cleaning
- 2. CyberAgent AI Lab MasashiSHIBATA c-bata c_bata_ Python
- 3.
- 4.
- 5. 1 2 3 4 Automated Feature Engineering AutoML AutomatedHyperparameter Optimization Automated Algorithm(Model) Selection
- 6.
- 7. Jeff Dean, “AnOverview of Google's Work on AutoML and Future Directions” , ICML 2019 https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions
- 8. Automated Hyperparameter Optimization Hyperopt,Optuna, SMAC3, scikit-optimize, … Jeff Dean, “An Overview of Google's Work on AutoML and Future Directions” , ICML 2019 https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions
- 9. Jeff Dean, “AnOverview of Google's Work on AutoML and Future Directions” , ICML 2019 https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions HPO + Automated Feature Engineering featuretools, tsfresh, boruta, …
- 10. Automated Algorithm(Model) Selection Auto-sklearn,TPOT, H2O, auto_ml, MLBox, … Jeff Dean, “An Overview of Google's Work on AutoML and Future Directions” , ICML 2019 https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions
- 11.
- 12. Grid Search /Random Search
- 14. Eric Brochu, VladM. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599.
- 15. Eric Brochu, VladM. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599. : :
- 16. Eric Brochu, VladM. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599.
- 17. Eric Brochu, VladM. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599.
- 18. Jamieson, K. G.and Talwalkar, A. S.: Non-stochastic Best Arm Identification and Hyperparameter Optimization, in AIS-TATS (2016). 10 epochs trial #1 trial #2 trial #3 trial #4 trial #5 trial #6 trial #7 trial #8 trial #9 30 epochs 90 epochs Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. arXiv preprint arXiv:1810.05934, 2018.
- 19.
- 23. TPE, Asynchronous SuccessiveHalving, Median Stopping Rule Define-by-Run https://github.com/pfnet/optuna
- 25. rung 0 0.088 0.0560.035 0.027 0.495 0.122 0.150 0.788 0.093 0.115 0.238 0.106 0.104 0.058 trial 0 trial 4 trial 2 trial 6 trial 24 trial 1 trial 8 trial 5 trial 18 trial 3 trial 7 rung 1 rung 2 rung 3 rung
- 26.
- 27.
- 28.
- 29. 1. Feature PreprocessingOperators. StandardScaler, RobustScaler, MinMaxScaler, MaxAbsScaler, RandomizedPCA, Binarizer, and PolynomialFeatures. 2. Feature Selection Operators: VarianceThreshold, SelectKBest, SelectPercentile, SelectFwe, and Recursive Feature Elimination (RFE). AutoML feature preprocessing M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In Neural Information Processing Systems (NIPS), 2015 R. S. Olson and J. H. Moore. Tpot: A tree-based pipeline optimization tool for automating machine learning. In Workshop on Automatic Machine Learning, 2016 TPOT Auto-sklearn
- 30. 1. Feature PreprocessingOperators. StandardScaler, RobustScaler, MinMaxScaler, MaxAbsScaler, RandomizedPCA, Binarizer, and PolynomialFeatures. 2. Feature Selection Operators: VarianceThreshold, SelectKBest, SelectPercentile, SelectFwe, and Recursive Feature Elimination (RFE). AutoML feature preprocessing M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In Neural Information Processing Systems (NIPS), 2015 R. S. Olson and J. H. Moore. Tpot: A tree-based pipeline optimization tool for automating machine learning. In Workshop on Automatic Machine Learning, 2016 TPOT Auto-sklearn
- 31. TPOT Auto-sklearn AutoML •featuretools: Deep feature synthesis • tsfresh Wrapper methods, Filter methods, Embedded methods • scikit-learn Boruta
- 32.
- 35.
- 36. 63 794 Time SeriesFeatuRE extraction based on Scalable Hypothesis tests https://github.com/blue-yonder/tsfresh
- 37.
- 38. Guyon and A.Elisseeff. An introduction to variable and feature selection. Journal of Machine Learning Research, 3:1157–1182, 2003.
- 39. Filter method Wrapper method sklearn.feature_selection.RFE(RecursiveFeature Elimination), Boruta (boruta_py) Embedded method scikit-learn feature_importances_ Guyon and A. Elisseeff. An introduction to variable and feature selection. Journal of Machine Learning Research, 3:1157–1182, 2003.
- 40.
- 41.
- 42. • AutoML 2 • ML • ML • AutoML as a CASH Problem Combined Algorithm Selection and Hyperparameter optimization M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. Efficient and robust automated machine learning. In Neural Information Processing Systems (NIPS), 2015
- 43. Using Optuna forCASH Problems def objective(trial): iris = sklearn.datasets.load_iris() x, y = iris.data, iris.target classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest']) if classifier_name == 'SVC': svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10) classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto') else: rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32)) classifier_obj = sklearn.ensemble.RandomForestClassifier( max_depth=rf_max_depth, n_estimators=10) score = sklearn.model_selection.cross_val_score( classifier_obj, x, y, n_jobs=-1, cv=3) accuracy = score.mean() return accuracy https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py
- 44. Optuna for CASHProblem def objective(trial): iris = sklearn.datasets.load_iris() x, y = iris.data, iris.target classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest']) if classifier_name == 'SVC': svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10) classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto') else: rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32)) classifier_obj = sklearn.ensemble.RandomForestClassifier( max_depth=rf_max_depth, n_estimators=10) score = sklearn.model_selection.cross_val_score( classifier_obj, x, y, n_jobs=-1, cv=3) accuracy = score.mean() return accuracy https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py Algorithm Selection
- 45. Optuna for CASHProblem def objective(trial): iris = sklearn.datasets.load_iris() x, y = iris.data, iris.target classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest']) if classifier_name == 'SVC': svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10) classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto') else: rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32)) classifier_obj = sklearn.ensemble.RandomForestClassifier( max_depth=rf_max_depth, n_estimators=10) score = sklearn.model_selection.cross_val_score( classifier_obj, x, y, n_jobs=-1, cv=3) accuracy = score.mean() return accuracy https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py Hyperparameter optimization
- 46. • auto-sklearn • TPOT •h2o-3 • auto_ml (unmaintained) • MLBox Adithya Balaji, Alexander Allen Benchmarking Automatic Machine Learning Frameworks https://arxiv.org/pdf/1808.06492v1.pdf
- 47. AutoML SMAC3 ChaLearn AutoML challenge2 track Auto-sklearn import sklearn.metrics import autosklearn.classification X_train, X_test, y_train, y_test = train_test_split(…) automl = autosklearn.classification.AutoSklearnClassifier(…) automl.fit(X_train.copy(), y_train.copy(), dataset_name='breast_cancer') print(automl.show_models()) predictions = automl.predict(X_test) print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions)) https://github.com/automl/auto-sklearn
- 48. TPOT https://github.com/EpistasisLab/tpot Tree-based Pipeline OptimizationTool for Automating Data Science
- 49. TPOT https://github.com/EpistasisLab/tpot from tpot importTPOTClassifier from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(…) tpot = TPOTClassifier(verbosity=2, max_time_mins=2) tpot.fit(X_train, y_train) print(tpot.score(X_test, y_test)) tpot.export('tpot_iris_pipeline.py')
- 50. • 20 preprocessors •16 feature selectors, • 1-hot encoding, missing value imputation, balancing, scaling • 17 classifiers • pre-defined hyperparameter spaces • 20 preprocessors • 12 classifiers • pre-defined hyperparameter spaces • Pipeline search space: • Flexible: combining tree-shaped pipelines Auto-sklearn TPOT
- 51.
- 52.
- 53.
- 54.
- 55.
- 57.
- 58.