DataSEA- Data Science, Engineering, Analytics

-
  # DataSEA: The Gamified Mobile App to Learn Data Engineering, SQL, and Analytics in 10 Minutes a Day                                                                                                                                                                                                             > **TL;DR** — DataSEA is a free Android app that turns Data Engineering, Analytics, and Data Science into bite-size, gamified lessons. 88+ modules,     500+ lessons,...
Post a Comment

Data Engineering Basics to Advance: Phase-IV- Capstone Projects

-

                        Capstone Projects


Project Ideas:

  1. Batch Pipeline

    • Source: CSV on S3

    • Process: PySpark/dbt

    • Sink: Redshift/BigQuery

    • Orchestrate: Airflow

  2. Streaming Pipeline

    • Source: Kafka (clickstream or logs)

    • Process: Spark Streaming

    • Sink: ElasticSearch or S3

  3. Data Observability

    • Implement Great Expectations

    • Data profiling and alerting

  4. Cloud-native Data Lakehouse

Comments

Post a Comment

Popular posts from this blog

Bhakti-Aarti- Android app Privacy policy

-
 Privacy Policy Effective Date: October 4th, 2025   App Name: Bhakti Aarti   Developer: k3kventures   Contact: sultanmirzadev@gmail.com --- 1. Introduction This Privacy Policy explains how we collect, use, and protect your personal information when you use our Android application "Bhakti Aarti" ("App"). By using the App, you agree to the terms of this Privacy Policy. --- 2. Information We Collect a. Voice Input Our App may request access to your device’s microphone in order to enable voice-based search functionality. When you use this feature: - The app listens only when you activate the voice input manually.   - Your voice input is processed to understand your search query.   - We do not store or share your voice recordings or voice data. Voice data may be processed by Google's speech recognition services (or similar services) if you are using a device with Google Play Services enabled. Please refer to Google’s Privacy Policy at:   https://policies.go...
Read more

DBT tool connect Athena from Local- AWS SSO

-
USE AWS SSO login to access Athena using DBT outputs : dev : type : athena s3_staging_dir : s3://test-bucket/athena/ region_name : ap-south-1 database : awsdatacatalog schema : <db_name>      work_group : < Athena-workgroup > aws_profile_name : <profile-name-on-local-machine>
Read more

AWS Lake formation - AWS LF - Governance Security- Access control

-
  🧭  Overview: What is AWS Lake Formation? AWS Lake Formation is a service that simplifies building a secure data lake by: Ingesting data from various sources Organizing it in Amazon S3 Setting up data catalogs (via AWS Glue) Defining security and access policies Querying data with services like Athena, Redshift, and EMR 🛠️  Prerequisites Before starting, ensure you have: An  AWS account IAM permissions for Lake Formation, Glue, S3, and IAM An existing  S3 bucket  (or create a new one) 🧱  Step 1: Set Up a Data Lake Location Go to the  Lake Formation Console . In the left pane, choose  "Data lake locations" . Click  "Register location" . Choose your  S3 bucket  or a folder (e.g.,  s3://your-bucket/data/ ). Choose an  IAM role  that has permission to access this location. 📋  Step 2: Add a Data Catalog Table From the Lake Formation Console, go to  "Databases" . Click  "Create database"  (this...
Read more