Are you looking to modernize your data infrastructure, reduce cloud costs, and build highly scalable ETL pipelines? I am a Senior Data Engineer with 3+ years of rigorous experience designing and implementing robust data architectures for leading tech organizations.
What I Offer:
- End-to-End ETL/ELT Pipelines: Designing, building, and deploying fault-tolerant batch and real-time streaming pipelines using Apache Spark (PySpark), Apache Kafka, and Airflow.
- Data Lake & Warehouse Architecture: Deep expertise in building modern data lakes and data warehouses on AWS (S3, Redshift), GCP, and Huawei Cloud (OBS), utilizing Delta Lake and Trino for highly scalable SQL analytics.
- Cloud Infrastructure & DevOps: Deploying and managing large-scale workloads on Kubernetes (K8s) and Docker. Automating infrastructure and CI/CD using Terraform, Ansible, and GitHub Actions.
- Performance Optimization & Cost Reduction: Proven track record of 'right-sizing' compute resources, tuning Spark applications, and reducing pipeline execution times by over 75% while slashing cloud costs by 50%.
- Data Quality & Observability: Building unified data quality frameworks to detect schema drift and volume anomalies, alongside robust alerting using Grafana and Metabase.
Why Choose Me (My Differentiators): Unlike standard back-end developers, my focus is purely on highly-optimized, enterprise-grade data platforms. I don't just write code; I architect solutions that decrease your infrastructure overhead and ensure 100% data integrity. My background includes leading a migration of over 500 complex pipelines with zero data loss, meaning you can trust me with your most mission-critical data migrations, scaling efforts, and deployments.
Core Skills & Keywords: Data Engineering, PySpark, Apache Airflow, Kafka, AWS, EC2, S3, Redshift, Kubernetes, K8s, Python, SQL, Databricks, Delta Lake, Snowflake, Trino, ETL, ELT, Data Pipelines, Cloud Migration, Performance Tuning.
Let's discuss how we can transform your data infrastructure into a reliable, cost-effective asset!