Designing data processing systemsSelecting the appropriate storage technologies. Considerations include:● Mapping storage systems to business requirements● Data modeling● Trade-offs involving latency, throughput, transactions● Distributed systems● Schema designDesigning data pipelines. Considerations include:● Data publishing and visualization (e.g., BigQuery)● Batch and streaming data (e.g., Dataflow, Dataproc, Apache Beam, Apache Spark and Hadoop ecosystem, Pub/Sub, Apache Kafka)● Online (interactive) vs. batch predictions● Job automation and orchestration (e.g., Cloud Composer)Designing a data processing solution. Considerations include:● Choice of infrastructure● System availability and fault tolerance● Use of distributed systems● Capacity planning● Hybrid cloud and edge computing● Architecture options (e.g., message brokers, message queues, middleware, service-oriented architecture, serverless functions)● At least once, in-order, and exactly once, etc., event processingMigrating data warehousing and data processing. Considerations include:● Awareness of current state and how to migrate a design to a future state● Migrating from on-premises to cloud (Data Transfer Service, Transfer Appliance, Cloud Networking)● Validating a migrationBuilding and operationalizing data processing systemsBuilding and operationalizing storage systems. Considerations include:● Effective use of managed services (Cloud Bigtable, Cloud Spanner, Cloud SQL, BigQuery, Cloud Storage, Datastore, Memorystore)● Storage costs and performance● Life cycle management of dataBuilding and o
What you'll learn
designing data processing systems
selecting appropriate storage technologies
mapping storage systems to business requirements
data modeling
designing data pipelines
implementing batch and streaming data solutions
building operational data processing systems
managing storage costs and performance
Course objectives
equip students with the skills needed to design and implement data processing solutions
prepare students for the Google Cloud Certified Professional Data Engineer exam
teach effective data pipeline construction and migration techniques