Door categorieën bladeren
Ontdekken
Fiverr Pro
Nederlands
$
USD
Vivek Pandey
@vivekpandey600
Senior Data Engineer
India
Hindi, Engels
Over mij
I am a Senior Data Engineer with over 6 years of experience designing and optimizing end-to-end pipelines on Azure Databricks and PySpark. I specialize in Medallion Architecture, real-time streaming with Kafka, and data governance using Unity Catalog for global brands in retail and energy.
Skills
Vivek Pandey
offline •
Bekijk mijn diensten
Data ETL's
I will be your databricks and pyspark data engineer for etl and delta lake pipelines
Werkervaring
Senior Associate Consultant
Infosys • Fulltime
Feb 2025 - Present • 1 yr 8 mos
Building Databricks Lakehouse pipelines for global retail finance and consumer-service data. Designing end-to-end batch ETL pipelines using Medallion Architecture with Delta Lake and Unity Catalog. Ingesting data from 1,200+ global stores into AWS S3. Implemented CDC for incremental loads and optimized PySpark transformations, reducing batch runtime by 35%. Managing Amazon Connect call-transcript pipelines processing 200K JSON files daily.
Azure Data Engineer
Tata Consultancy Services • Fulltime
Feb 2020 - Jan 2025 • 4 yrs 11 mos
Data Engineer for TotalEnergies' digital drilling operations. Built real-time streaming pipelines using Kafka and PySpark to ingest sensor and rig performance data. Processed 1 TB/day of sensor data into Azure Data Lake and Cosmos DB. Reduced pipeline execution time by 50% through performance optimization. Developed batch pipelines with Azure Function Apps and automated job monitoring, saving 2 hours of manual effort daily.
