Download as PDF, PPTX
![record_transformer copy file BigQuery
<filter frontend.user.*>
@type record_transformer
enable_ruby
remove_keys host
<record>
bq {"insert_id":"${uid}","host":"${host}",
"created":"${time.to_i}"}
avg ${record["total"] / record["count"]}
</record>
</filter>
syntax: Ruby, easy to use.
Great for:
- date transformation,
- quick normalizations,
- calculating something on the fly,
and store in clear log/analytics db
- renaming without code deploy.
1 2 3 4
GCP Solutions for DevOps Engineers @martonkodok](https://image.slidesharecdn.com/devopsproeurope-190321090159/85/Google-Cloud-Platform-Solutions-for-DevOps-Engineers-48-320.jpg)
![record_transformer copy file BigQuery
<filter frontend.user.*>
@type record_transformer
enable_ruby
remove_keys host
<record>
bq {"insert_id":"${uid}","host":"${host}",
"created":"${time.to_i}"}
avg ${record["total"] / record["count"]}
</record>
</filter>
syntax: Ruby, easy to use.
Great for:
- date transformation,
- quick normalizations,
- calculating something on the fly,
and store in clear log/analytics db
- renaming without code deploy.
1 2 3 4
GCP Solutions for DevOps Engineers @martonkodok](https://image.slidesharecdn.com/devopsproeurope-190321090159/75/Google-Cloud-Platform-Solutions-for-DevOps-Engineers-48-2048.jpg)
Uploaded byMárton Kodok
Google Cloud Platform Solutions for DevOps Engineers
The document outlines Google Cloud Platform (GCP) solutions for DevOps engineers, emphasizing seamless application development and serverless computing. It discusses various services like App Engine, Cloud Functions, and BigQuery, highlighting their capabilities in scalability, cost efficiency, and integration for cloud applications. Additionally, it covers monitoring tools like Google Stackdriver and practical use cases for hybrid cloud architectures.
In this document
Powered by AI
Introduction of Márton Kodok, his credentials, and background in GCP as a developer expert.
Outline of the presentation agenda covering application development, cloud services, and practical use cases.
Identifies difficulties in traditional app development such as IaaS, OS setup, storage, and container orchestration.
Highlights that REEA.net operates on the same robust infrastructure utilized by Google for their services.
Detailed listing of GCP services including computing, data storage, machine learning, security, and more.
Serverless architecture focus, emphasizing ease of use, cost management, and a shift in operational models.
App Engine explained as a managed platform for seamless scaling and traffic management.
Describes async message processing and task queues using Cloud Tasks and its efficiency in API workloads.
Introduction to Cloud Functions as a GCP service focusing on event-driven functions and service integration.
Details on various events that can trigger Cloud Functions including Cloud Storage and BigQuery actions.
Comparative discussion on when to use App Engine versus Cloud Functions based on application needs.
Encourages flexible application structuring to enhance agility and delivery of applications.
Discussion on in-house versus hosted monitoring solutions for efficient service management.
Explains the features and benefits of Google Stackdriver for monitoring and managing applications.
Overview of BigQuery as a cloud-based data warehouse capable of handling large datasets.
Discusses integration and processing capabilities of BigQuery for comprehensive data analysis.
Details on utilizing FluentD with BigQuery for data transformation and integration.
Outlines advantages of BigQuery including cost efficiency, powerful querying capabilities, and ease of use.
Information on available Qwiklabs to enhance learning about GCP services and practical applications.
Thank you note with links to further resources and slides for presentation access.
More Related Content
Similar to Google Cloud Platform Solutions for DevOps Engineers
Google Cloud Platform Solutions for DevOps Engineers
- 1. GoogleCloudPlatformSolutions forDevOpsEngineers Márton Kodok /@martonkodok Google Developer Expert at REEA.net - Targu Mures March 2019 - Vilnius, Lithuania
- 2. ● Geek. Hiker.Do-er. ● Among the Top3 romanians on Stackoverflow 130k reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery/Redis and database engine expert ● Active in mentoring and IT community StackOverflow: pentium10 GitHub: pentium10 Slideshare: martonkodok Twitter: @martonkodok GCP Solutions for DevOps Engineers @martonkodok About me
- 3. 1. Application developmentin the Cloud 2. App Engine: Scale your apps seamlessly from zero to planet scale 3. Cloud Functions: your gateway to GCP Services 4. Google Stackdriver: Metrics, logging, alerting are a universal right! 5. BigQuery: federated data access warehouse 6. Doing hybrid cloud mixing on premise with cloud 7. Practical use cases 8. Qwiklabs Agenda GCP Solutions for DevOps Engineers @martonkodok
- 4. What’s so hardabout traditional app development? GCP Solutions for DevOps Engineers @martonkodok User Select IaaS OS Setup Containerization Storage Container Orchestration Networking Monitoring Logging Testing Dev
- 5. REEA.net uses GCP Buildon the same infrastructure that powers Google
- 6. Google Cloud Platform(GCP) Compute Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Storage & Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Cloud Spanner Persistent Disk Machine Learning Cloud Machine Learning Cloud Vision API Cloud Speech API Cloud Natural Language API Cloud Translation API Cloud Jobs API Data Studio Cloud Dataprep Cloud Video Intelligence API Advanced Solutions Lab Compute Engine App Engine Kubernetes Engine GPU Cloud Functions Container- Optimized OS Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Key Management Service BeyondCorp Data Loss Prevention API Identity-Aware Proxy Security Key Enforcement Internet of Things Cloud IoT Core Transfer Appliance
- 7. Google Cloud Platform(GCP) Developer Tools Cloud SDK Cloud Deployment Manager Cloud Source Repositories Cloud Tools for Android Studio Cloud Tools for IntelliJ Cloud Tools for PowerShell Cloud Tools for Visual Studio Container Registry Google Plug-in for Eclipse Cloud Test Lab Networking Virtual Private Cloud Cloud Load Balancing Cloud CDN Cloud Interconnect Cloud DNS Cloud Network Cloud External IP Addresses Cloud Firewall Rules Cloud Routes Cloud VPN Cloud Router Dedicated Interconnect Container Builder Management Tools Stackdriver Monitoring Logging Error Reporting Trace Debugger Cloud Deployment Manager Cloud Console Cloud Shell Cloud Mobile App Cloud Billing API Cloud APIs Profiler
- 8. Google sees serverlessas GCP Solutions for DevOps Engineers @martonkodok Programming model Focus on code Event-driven Stateless Operational model Billing model Pay for usageZero ops Automatic scaling Managed security Dev Ops $
- 9. Serverless is morethan a set of functions GCP Solutions for DevOps Engineers @martonkodok Cloud Dataflow Cloud Tasks Cloud Storage Cloud PubSub Cloud Functions App Engine BigQuery Stackdriver
- 10. Meet Serverless serverless datacenter depicted GCP Solutions for DevOps Engineers @martonkodok
- 11. Serverless is aboutmaximizing elasticity, cost savings, and agility of cloud computing. GCP Solutions for DevOps Engineers @martonkodok
- 12. Serverless types GCP Solutionsfor DevOps Engineers @martonkodok Platforms Triggered Code
- 13. App Engine -managed application platform GCP Solutions for DevOps Engineers @martonkodok API 2 Compute Engine Virtual Machines API 1 App Engine Split App Versions API 3 Container Engine Kubernetes Services Multiple Clients API Requests Cloud Load Balancing ● Scale your applications seamlessly from zero to planet scale ● Automatically scales depending on your application traffic ● Traffic Splitting (app versions, A/B tests, incremental rollouts) Best used: ● For HTTP services ● For existing applications
- 14. App Engine: Servicesand versions GCP Solutions for DevOps Engineers @martonkodok Application Version Service Version Instance Instance Service Version Version Instance Instance
- 15. App Engine: Trafficsplitting GCP Solutions for DevOps Engineers @martonkodok A/B testing and Canary releasing with a few clicks or a single gcloud command
- 16. App Engine: Runtimes GCPSolutions for DevOps Engineers @martonkodok Java 8 1.11
- 17. GCP Solutions forDevOps Engineers @martonkodok Async message processing
- 18. Asynchronous task execution GCPSolutions for DevOps Engineers @martonkodok Cloud Tasks Cloud PubSub Message queue system Guaranteed at-least-once delivery Future Scheduling Will only be dispatched once on dups Best used: ● For HTTP services ● For App Engine handlers Deliver event data based on topics Exactly-once processing Build multi-cloud on premise, hybrid apps Cross zone message replication Best used: ● For large-scale ingestion of events, streams ● Topics, publish/subscribe patterns, IoT
- 19. Cloud Tasks -Message Queue systems GCP Solutions for DevOps Engineers @martonkodok Workers Compute Engine Static Content Cloud Storage Dynamic Content Cloud SQL Dynamic Content Cloud Datastore Front End App App Engine Autoscaling Cloud Tasks Memcache Batch App App Engine Autoscaling Cloud Load Balancing Cloud DNS Workers Compute Engine Autoscaling
- 20. Task Queues GCP Solutionsfor DevOps Engineers @martonkodok Slides: https://www.slideshare.net/martonkodok Title: Architectural Patterns - Message Queues Push queues ● Put with delays Pull queues ● Ability to “tag” ● Lease multiple eg: gameboard updated, game id as tag.
- 21. Reduce request latency GCPSolutions for DevOps Engineers @martonkodok
- 22. GCP Solutions forDevOps Engineers @martonkodok Cloud Functions
- 23. Cloud Functions ApplicationEvent Sourcing Frontend PlatformServices Metrics / Logs/ Streaming Event Triggered Cloud Functions Triggered Code GCP Solutions for DevOps Engineers @martonkodok Result
- 24. GCP Solutions forDevOps Engineers @martonkodok Unit: Function Trigger: Events and HTTP Best used: ● For Events & Async workloads ● For single-purpose microservices Cloud Functions
- 25. Cloud Functions -Event-driven - FaaS GCP Solutions for DevOps Engineers @martonkodok ● Simplest way to run your code in the cloud - just deploy ● Automatically scales, highly available and fault tolerant ● No servers to provision, manage, patch or update ● Pay only while your code runs ● Connects and extends cloud services (In alpha: Java) ● Node 8.11 ● Support for async/await ● Python 3.7.1 ● Flask microframework ● Go 1.11 ● Familiar building blocks
- 26. Functions: your gatewayto GCP Services GCP Solutions for DevOps Engineers @martonkodok Access 20+ Google services from GCF BigQuery Cloud Pub/Sub Cloud Storage Cloud Bigtable Cloud Datastore Cloud Spanner Cloud Machine Learning Cloud Vision API Cloud Speech API Cloud Natural Language API Cloud Translation API Stackdriver Services Applications Firebase APIs Assistant
- 27. Cloud Functions Trigger GCPSolutions for DevOps Engineers @martonkodok Cloud Storage Finalize/Create Delete Archive Metadata Update Cloud Pub/Sub Topic Compute Engine EAP Instance Instance Group Autoscaler Firewall Set Labels
- 28. Cloud Functions Trigger GCPSolutions for DevOps Engineers @martonkodok BigQuery Job create Job complete Firebase Realtime DB change Authentication Remote config Google Analytics log Cloud Firestore Create Update Write Delete
- 29. Summary: When touse what GCP Solutions for DevOps Engineers @martonkodok App Engine Unit of computing is apps HTTP request/response Large scalable backends Cloud Functions Smallest unit of computing Event driven architecture Connect & extend services Serverless add-on Run functions, apps & containers on GKE Full portability of your artifacts Run on your own cluster
- 30. Slice and diceyour application any way you think it makes sense to be more agile, to build better apps that deliver value. The unit of computing is yours to chose! GCP Solutions for DevOps Engineers @martonkodok
- 31. GCP Solutions forDevOps Engineers @martonkodok How I Learned How to Stop Worrying and Dig Hosted Monitoring
- 32. GCP Solutions forDevOps Engineers @martonkodok What to do for monitoring? ● In-house or hosted? ● Modern vs proven? ● Specialized or general? ● Paid vs free? Let’s be honest: ● Not enough time ● Small environment ● More important work to do ● Everyone else is doing it
- 33. GCP Solutions forDevOps Engineers @martonkodok Intelligent monitoring and management for services, containers, applications, and infrastructure. ● Works with GCP, AWS, on prem ● Logging, tracing, alerts ● Collectd agent ● Cost per resource ● Easy point and click alerting policy ● Debugger, Error reporting, profiler *free ● Uptime checks *free Google Stackdriver
- 34. GCP Solutions forDevOps Engineers @martonkodok Google Stackdriver: Uptime check (offered free) ● Check Type: HTTP, TCP ● Resource Type: URL, Instance, App Engine, Load Balancer ● Global Locations ● Custom Headers (encrypted) ● Authentication ● Whitelisted source IPs ● Alerting policy: Email, SMS, Slack, PagerDuty, Hipchat, Campfire, Webhooks
- 35. GCP Solutions forDevOps Engineers @martonkodok Google Stackdriver: Uptime check
- 36. GCP Solutions forDevOps Engineers @martonkodok Life of a log
- 37. GCP Solutions forDevOps Engineers @martonkodok Google Stackdriver Metrics, logging, alerting are a universal right!
- 38. GCP Solutions forDevOps Engineers @martonkodok
- 39. Analytics-as-a-Service - DataWarehouse in the Cloud Scales into Petabytes on Managed Infrastructure - load up to 5TB large files SQL 2011 + Javascript UDF (User Defined Functions) Familiar DB Structure (table, columns, views, struct, nested, JSON) Integrates with Google Sheets + Cloud Storage + Pub/Sub connectors Decent pricing (storage: $20/TB cold: $10/TB,queries $5/TB) *March 2019 What is BigQuery? GCP Solutions for DevOps Engineers @martonkodok
- 40. Architecting for TheCloud BigQuery On-Premises Servers Pipelines ETL Engine Event Sourcing Frontend Platform Services Metrics / Logs/ Streaming GCP Solutions for DevOps Engineers @martonkodok
- 41. BigQuery: federated dataaccess warehouse GCP Solutions for DevOps Engineers @martonkodok Application & Presentation Audit logs Billing entries Stackdriver Firebase Google Marketing Platform Cloud Dataflow Cloud Storage Report & Share Business Analysis BI Interface Data Studio 360 Analysis Processing ML Frontend Platform Services Real-Time Events Multiple Platforms Database SQL
- 42. “ Data needsto be processed in multiple services. How can we pipe to multiple places? GCP Solutions for DevOps Engineers @martonkodok
- 43. Architecting for TheCloud On-Premises Servers Event Sourcing Frontend Platform Services Analyze Metrics / Logs/ Streaming Cloud Storage GCP Solutions for DevOps Engineers @martonkodok Cloud Dataflow Process BigQuery Cloud SQL Stream Batch Data Studio Third-Party Tools
- 44. “ We haveour app outside of GCP. How can we use the benefits of BigQuery? GCP Solutions for DevOps Engineers @martonkodok
- 45. Data Pipeline Integrationat REEA.net Analytics Backend BigQuery On-Premises Servers Pipelines FluentD Event Sourcing Frontend Platform Services Metrics / Logs/ Streaming Development Team Report & Share Business Analysis Tools Tableau QlikView Data Studio Internal Dashboard Database SQL Application ServersServers Cloud Storage archive Load / Export Replay Standard Devices HTTPS GCP Solutions for DevOps Engineers @martonkodok Cloud Functions
- 46. The following slideswill present a sample Fluentd configuration to: 1. Transform a record 2. Copy event to multiple outputs 3. Store event data in File (for backup/log purposes) 4. Stream to BigQuery (for immediate analyses) GCP Solutions for DevOps Engineers @martonkodok
- 47. <filter frontend.user.*> @type record_transformer </filter> <matchfrontend.user.*> @type copy <store> @type forest subtype file </store> <store> @type bigquery </store> … </match> Filter plugin mutates incoming data. Add/modify/delete event data transform attributes without a code deploy.1 2 3 4 The copy output plugin copies events to multiple outputs. File(s), multiple databases, DB engines. Great to ship same event to multiple subsystems. The Bigquery output plugin on the fly streams the event to the BigQuery warehouse. No need to write integration. Data is available immediately for querying. Whenever needed other output plugins can be wired in: Kafka, Google Cloud Storage output plugin. GCP Solutions for DevOps Engineers @martonkodok
- 48. record_transformer copy fileBigQuery <filter frontend.user.*> @type record_transformer enable_ruby remove_keys host <record> bq {"insert_id":"${uid}","host":"${host}", "created":"${time.to_i}"} avg ${record["total"] / record["count"]} </record> </filter> syntax: Ruby, easy to use. Great for: - date transformation, - quick normalizations, - calculating something on the fly, and store in clear log/analytics db - renaming without code deploy. 1 2 3 4 GCP Solutions for DevOps Engineers @martonkodok
- 49. record_transformer copy fileBigQuery <match frontend.user.*> @type copy <store> @type forest subtype file <template> path /tank/storage/${tag}.*.log time_slice_format %Y%m%d </template> </store> </match> 1 2 3 4 GCP Solutions for DevOps Engineers @martonkodok
- 50. record_transformer copy fileBigQuery <match frontend.user.*> @type bigquery method insert auth_method json_key json_key /etc/td-agent/keys/key-31da042be48c.json time_field timestamp time_slice_format %Y%m%d table user$%{time_slice} ignore_unknown_values schema_path /etc/td-agent/schema/user_login.json </match> 1 2 3 4 Connector uses: - JSON key auth file - JSON table schema Pro features: - streaming to Partitioned tables - ignore unknown values (not reflected in schema) GCP Solutions for DevOps Engineers @martonkodok
- 51. ● SQL+JSON (bigcosts saving with partitioning/clustering) ● run raw ad-hoc queries (either by analysts/sales or Devs) ● ability to throw in / join all kind of data ● pricing model 1TB free every month ● no more throwing away-, expiring-, aggregating old data ● no provisioning/deploy ● no running out of resources ● no more focus on large scale execution plan Our benefits using BigQuery GCP Solutions for DevOps Engineers @martonkodok
- 52. Kubernetes Machine LearningCertification Introductory For GCP newcomers 1 hour duration, 2 labs Enroll in the Baseline: Infrastructure Quest. 1. Kubernetes Engine: Qwik Start (lab) (video) - Deploy a containerized application with Kubernetes Engine. 2. IoT: Qwik Start (lab) - Learn about GCP’s Cloud IoT Core service, where almost any IoThing is possible! Enroll in the Baseline: Data, ML, AI Quest. 1. Cloud ML Engine: Qwik Start (lab) (video) - Train and deploy a TensorFlow model to Cloud ML Engine. 2. Cloud Natural Language API: Qwik Start (lab) - Perform sentiment analysis on a block of text. For fun, paste in the last email you sent! Enroll in the Cloud Architecture Quest. 1. Stackdriver Qwik Start: Web UI (lab) What’s up with your VM? Learn to install monitoring and logging agents to collect info about your GCP resources. 2. Getting Started with Cloud KMS (lab) - Create your first KeyRing and CryptoKey with GCP. Experienced Qwiklabs alumni 1 hour, 2 labs Enroll in the Kubernetes Quest. 1. Kubernetes Engine: Qwik Start (lab) (video) - Deploy a containerized application with Kubernetes Engine. 2. Intro to Docker (lab) Are your containers running? Create, run, and debug containers, then pull and push images to and from Google Container Registry. Enroll in the Machine Learning APIs Quest. 1. Cloud ML Engine: Qwik Start (lab) (video) - Train and deploy a TensorFlow model to Cloud ML Engine. 2. Detect Labels, Faces, and Landmarks in Images with the Cloud Vision API (lab) - How do you feel about self-driving cars? Learn what’s behind the AI making the future possible. Enroll in the Cloud Architecture Quest. 1. Stackdriver Qwik Start: Web UI (lab) - What’s up with your VM? Learn to install monitoring and logging agents to collect info about your GCP resources. 2. Application Performance Management (APM) with Stackdriver (lab) - Dig deeper into Stackdriver with a problematic environment - can you identify the issues? Qwiklabs - Find Your Topic and Skill Level GCP Solutions for DevOps Engineers @martonkodok
- 53. Thank you. Slides availableon: slideshare.net/martonkodok Reea.net - Integrated web solutions driven by creativity to deliver projects.