YYash Gupta
Data Engineer · Bangalore, India

Building data platforms that are fast, reliable & cheap to run.

I independently own and scale data platforms in fast-paced, lean environments — leading critical migrations and architectural shifts that cut query latency by 90%+ and infrastructure cost by 35–40%. I specialize in cost-efficient, reliable pipelines and real-time systems.

Working across
ClickHouse
Spark
Airflow
AWS
dbt
Data Platform Orchestrator
··:··:··
Postgres
Postgres
PeerDB · CDC
PeerDB · CDC
ClickHouse
ClickHouse
Query SLA
60s5s
↓ 12× faster
Throughput
1.24M
▲ rows / sec
Ingest throughputlive
medallion · bronze → silver → gold synced
300GB DocumentDB → Aurora · 0 downtime
dbt run · 124 models building…
0%
Query latency reduced
60s5s
0%
AWS Aurora cost cut
100%60%
0GB
Migrated, zero downtime
100% uptime0 drops
0%
Lakehouse cost reduction
100%65%
01 — About

Strong ownership in lean engineering environments.

I'm a Data Engineer who likes the hard, unglamorous problems: migrations that can't drop a row, warehouses that need to answer in seconds instead of minutes, and cloud bills that need to come down without anyone noticing a regression.

⚙️

Platform migrations

Moved a cloud-native lakehouse to ClickHouse and 300GB from DocumentDB to Aurora Postgres — structured, zero-downtime, no cost increase.

📉

Cost engineering

Consolidated databases and re-architected the medallion lakehouse to drive 35–40% cost reductions across production and lower environments.

📡

Real-time & reliability

Built real-time monitoring dashboards with automated alerting for proactive issue detection and higher data reliability.

02 — Experience

Where I've shipped.

Lokal

Data Engineer

Lokal
Bengaluru, India
Jul 2026 — Present

India's largest local-language content, news & community platform (YC S19).

Connect and Heal

Data Engineer I

Connect and Heal
Bangalore, India
Jun 2024 — Jun 2026

Healthcare technology company focused on digital solutions to improve patient engagement and management.

0×faster queries0%cost cut0GBmigrated
  • Migrated the data platform from a cloud-native lakehouse to ClickHouse, optimizing warehouse design and cutting query SLA from 60+s to ~5s on average.
  • Consolidated and migrated databases, reducing AWS Aurora Postgres costs across production and lower environments by 40%.
  • Designed scalable workflows with S3, MWAA, Glue, DMS, Hudi, Spark & EMR in a medallion architecture — driving a 35% lakehouse cost reduction.
MagicPin

Software Engineer Intern

MagicPin
Gurugram, India
May 2022 — Aug 2022

A tech startup focused on enhancing retail & shopping experiences through data-driven insights.

0%faster crawl0%smaller images
  • Enhanced data crawling & parsing to efficiently extract product information from multiple sources — a 10% improvement in speed and performance.
  • Applied Python web crawling to automate data collection, improving efficiency and scalability by reducing Docker image size by 20%.
  • Assisted in deploying containerized data pipelines using Docker & Kubernetes, improving the scalability of data ingestion workflows.
National University of Singapore

Summer Research Intern

National University of Singapore
Singapore
Jun 2022

A leading research university specializing in innovation and advanced technology.

0ndplace · innovation
  • Improved AI model performance through optimization and experimental evaluation, in collaboration with NUS and Hewlett Packard Enterprise.
  • Group research project: designed an AI-powered fashion design system using Neural Style Transfer (NST) and Generative Adversarial Networks (GANs).
03 — Selected work

The migrations behind the numbers.

Platform migration

Cloud lakehouse → ClickHouse

60s5s12× faster
avg query SLA

Analytical queries were taking 60+ seconds, throttling dashboards and every downstream consumer.

Re-modeled the warehouse on ClickHouse — table engines, sort keys and materialized rollups tuned for the access patterns that actually mattered.

ClickHouseSparkEMRS3
Zero-downtime migration

300GB DocumentDB → Aurora

NoSQLSQL0 downtime
300GB migrated

Unstructured documents in DocumentDB blocked relational analytics and carried rising cost.

Modeled the documents into a relational schema and moved 300GB via DMS with CDC — a live cutover with no downtime and no cost increase.

DMSAuroraDocumentDBPython
Cost engineering

Medallion lakehouse re-architecture

100%60%−35–40%
infra spend

Aurora and lakehouse spend was climbing across production and lower environments.

Consolidated databases and re-architected the bronze → silver → gold lakehouse with S3, Glue, Hudi, Spark & MWAA.

HudiGlueAirflowS3
04 — Architecture

How a medallion lakehouse flows.

Gold·Serving layer

Curated marts and rollups served from ClickHouse — the sub-5-second layer that powers dashboards, alerting and every downstream consumer.

ClickHouseAirflowdbt
05 — Stack

The tools I reach for.

🧮

Data Processing & Platforms

Modeling warehouses and crunching billions of rows.

Spark
SQL
Python
ClickHouse
Hudi
Iceberg
☁️

Cloud & Storage

The AWS backbone every pipeline runs on.

S3
EMR
Glue
Lambda
RDS
DynamoDB
Athena
DMS
IAM
EC2
ECS
ECR
DocumentDB
Aurora Postgres
🪄

Orchestration & Transformation

Scheduling, lineage, and clean transforms.

Airflow (MWAA)
Dagster
dbt
🛠️

DevOps & Tools

Shipping, automating, and pairing with AI.

Docker
Jenkins
Git
Claude
🧪

Explored · PoC

Evaluated in proofs-of-concept and spikes.

Redshift
RisingWave
OLake
PeerDB
06 — Education

Where it started.

Shiv Nadar University

B.Tech in Computer Science Engineering

Shiv Nadar University · India

Aug 2020 — May 2024
07 — Projects

Things I've built for fun.

Airflow
Spark
MinIO
Metabase

Stock Market Data Pipeline

End-to-end pipeline on Apache Airflow + Docker to ingest, process & store daily stock data. Dockerized Spark transforms, stored in MinIO (S3-compatible) & PostgreSQL, visualized in Metabase.

AirflowDockerSparkMinIOPostgreSQLMetabase
style+contentdesign

AI-Generated Fashion Design

AI design system using Neural Style Transfer & GANs to blend artistic styles into unique patterns — reducing iteration time from hours to seconds. Secured 2nd place at NUS for innovation.

PythonNSTGANsDeep Learning
08 — Beyond the terminal

What I do when I'm not building pipelines.

FinTech & Markets

FinTech & MarketsMarket microstructure and trading infrastructure.

🏊

SwimmingEarly-morning laps — my no-notifications reset button.

Football

FootballA fresh Premier League convert — still picking my club.

Chess

ChessEndgame-obsessed; I over-study openings and still blunder.

Gaming

GamingStrategy sims to co-op shooters, streamed to nobody.

Discord, basically my OS

Discord, basically my OSVoice channels, listening parties, and a second-brain server.

09 — Contact

Let's build something reliable.

Open to data engineering roles & interesting platform problems. The fastest way to reach me is email.

Tap to copy · or compose in Gmail · Mail app

LinkedIn·+91 99901 81300·Bangalore, India
10 — What's next

More is on the way.

✍️

Blog & writing

Coming soon

Deep dives on ClickHouse tuning, cost engineering, and zero-downtime lakehouse migrations.

🔍

Project case studies

Coming soon

Architecture diagrams, the trade-offs I weighed, and the metrics behind each build.

🧪

Experiments & PoCs

Coming soon

Field notes from RisingWave, Dagster, dbt & Olake — streaming and orchestration evaluations.

Live demos

Coming soon

Interactive walkthroughs of real-time monitoring and data-quality pipelines.