Data Platform / Lakehouse Engineer Profiles

Architected and implemented a fully private modular AWS data platform using TypeScript CDK IaC. Built real-time serverless telemetry ingestion pipeline processing 100 million events daily. Led modernization of legacy data warehouse to a serverless Iceberg lakehouse, enabling near real-time analytics for 100M+ daily records. Automated provisioning and deployment via GitHub Actions OIDC with containerized Airflow images. Implemented dynamic retention/tiering with Glacier, reducing S3 costs to sub-$1K/month while retaining analytical accessibility.

Architect and scale enterprise-grade, cloud-native data platforms on AWS, GCP, and Azure, leveraging Snowflake, BigQuery, dbt, Airflow, and Python for real-time streaming and batch ETL/ELT pipelines. Reduced data latency by 40% and improved system availability by 35%. Reduced MTTR by 30% through proactive alerting and monitoring. CI/CD automation using Azure DevOps, AWS CodePipeline, GitHub, and CloudWatch. Also built MLOps pipelines with Kubernetes and Kubeflow.

Azure Data Factory + Synapse pipelines that cut data-integration time 45%. Re-engineered high-volume Databricks ETL frameworks processing billions of rows daily, reducing runtime 50% and increasing throughput 100×. Migrated PostgreSQL/Redshift to Synapse + Data Lake, achieving 35% cost reduction and 200% storage scalability. Implemented Microsoft Fabric–powered monitoring. Automated infrastructure with Terraform, ARM & Bicep, reducing management cost 90%. VP at Charles River (Aug 2022–Present).

Built loosely-coupled data lake and microservice infrastructure using Glue, Spark, Lambda, Batch, Step Functions, Athena, DBT — enabling inter-team collaboration. Rebuilt technical debt as IaC with CI/CD, increasing team velocity 2x. Migrated multiple TB of data, improving development speed cycle by ~50%. Prior SRE experience at Lumin Digital — maintained high security standards, improved workflows with Argo CD, Argo Workflows. SRE Team at Lumin before founding data engineer role.

Owns end-to-end lakehouse ingestion for risk, security, and compliance analytics at Vanguard across 20+ enterprise sources, 50+ automated control checks, and 90+ files under daily SLA enforcement. Optimized Apache Iceberg table properties for high-concurrency ingestion. Led migration of Dremio products to Iceberg lakehouse tables. Built standardized DQ framework using AWS Glue and Step Functions. Led migration of 100+ repositories from Bitbucket to GitHub with CI/CD pipelines via CloudFormation.

Developed end-to-end ML pipelines on Databricks with PySpark and Delta Lake. Developed a Universal Data Layer (UDL) leveraging AWS S3, Apache Iceberg, and AWS Glue Catalog for zero-copy data sharing. Developed features for Self-Serve Data Observability Service — automating Data Quality, Data Recon, Data Drift, and Schema Drift configuration. Developed CI/CD pipelines for DBT models. Prior experience at Cognizant (2017–2021) building Spark streaming with Kafka and Snowflake.

Lead Data Engineer delivering production-grade data platforms on AWS using Databricks/Spark/Delta, Snowflake, dbt, Airflow, and Kafka. Owns platform standards for ingestion, transformation, orchestration, testing, monitoring, and release readiness. Builds and maintains Airflow DAGs on MWAA with reliable retry strategy, idempotent patterns, and safe backfills. Develops Databricks Spark pipelines for batch and near real-time processing; optimizes Delta tables. Currently at Amazon (Jul 2024–Present). Previously at AT&T and Microsoft.

Staff-level Platform Engineer and SRE with 20+ years of experience designing, scaling, and operating mission-critical cloud and data platforms. Specialized in AWS, Site Reliability Engineering (SLOs, SLIs, incident management), Databricks, Delta Lake, and OneLake. Leading platform engineering and SRE efforts at Capital One. Previously at Wipro for 15 years. Very strong profile but 20+ years total experience — outside criteria.