1
Module 7 In this module we will:
• Query from External Data Sources
• Avoid Data Ingesting Pitfalls
Ingesting New Datasets
• Ingest New Data into Permanent Tables
into Google BigQuery • Discuss Streaming Inserts
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
2
Ingest data permanently into BigQuery from a variety of formats
Cloud Google
Storage Drive
Cloud BigQuery
Cloud Query
Dataflow Dataprep Engine
BigQuery Managed Storage
CSV,
Data ingested into BigQuery is
JSON,
Cloud stored in permanent tables and
Avro
Bigtable this storage is scalable and
fully-managed
… and more formats
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
3
BigQuery can query external data sources in GCS and Drive directly
Cloud Google
Storage Drive
Cloud BigQuery
Cloud Query
Dataflow Dataprep Engine
BigQuery Managed Storage
CSV,
JSON,
Cloud AVRO You can query external data
Bigtable sources directly from BigQuery
which bypasses managed
storage
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
4
Pitfalls: Querying from External Data Sources Directly
Limitations
● Strong Performance
Disadvantages
● Data Consistency not
Guaranteed
● Can’t Use Table Wildcards
(cool feature we will
introduce shortly)
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
5
Streaming records into BigQuery through the API
BigQuery
Query
Engine
Streaming Record Inserts BigQuery Managed Storage
Streaming data allows you to
query data without waiting for a
full batch load
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
6
Summary: GCPnew
Summary: Ingest offers youinto
datasets theBigQuery
abilitymanaged
to: storage
Data stored Load new data Setup streaming
permanently in from a variety of ingestion into
BigQuery is formats BigQuery through
fully-managed APIs
(performance,
backups,
redundancy).
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
7
Lab 6
Ingesting and Querying
New Datasets
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
Ingesting and Querying New Datasets
In this lab, you will ingest new data
sources into Google BigQuery and learn
how to query external data sources
directly.
© 2017 Google Inc. All rights reserved. Google and the Google logo are trademarks of Google Inc. All other
company and product names may be trademarks of the respective companies with which they are associated.
