Professional Experience

Over 8 years of experience in data engineering, backend development, and building scalable systems

Senior Data Engineer (Senior Manager)

Mediacorp Pte. Ltd.

Singapore

Sep 2023 - Present

A leading media company in Southeast Asia

  • Content Recommendation & Reliability: Architected and implemented a robust fallback recommendation service using Python and Flask, leveraging RESTful API best practices to significantly enhance system availability and reliability across Mediacorp's media properties
  • Backend Pipelines & Analytics: Designed and built end-to-end data pipelines to generate trending content recommendations and developed a scalable data ingestion platform for Wondershop's order analytics, enabling actionable insights for the analytics team
  • Innovative Features: Integrated advanced summarisation and translation capabilities for daily articles using LLM, and developed a chatbot with Azure Open AI studio to deliver in-depth, descriptive analysis of all major Mediacorp contents and websites performance
  • Cost & Efficiency Optimization: Re-architected critical data pipelines and optimized data models including JAVA-based backend components, to reduce compute costs and boost system efficiency
  • Custom Big Data Solutions: Collaborated in establishing a custom Spark environment with Jupyter Notebooks and Airflow, providing a cost-effective alternative to Databricks for improved data processing
Technologies:
Python Flask Azure Cloud Azure OpenAI Studio Langchain Spark Airflow Databricks Java LLMs

Software Development Engineer II

Flipkart

Bengaluru, India

Oct 2018 - Jul 2023

Leading e-commerce platform in India

  • Platform Modernization: Migrated key recommendation pipelines from Hadoop to Spark, redesigning user-centric systems to enhance scalability and performance on a leading e-commerce platform
  • Efficient Processing: Engineered a sophisticated post-order replenishment pipeline with dynamic restocking logic, and developed "HawkEye"—a computation layer using Spark SQL and Parquet—to drastically reduce HDFS read times
  • Performance Improvements: Optimized the user events enrichment process for 13TB of data, reducing processing time from 24 hours to 4.5 hours, while implementing incremental batch designs that cut runtime by 70% and resource usage by 80%
  • Real-Time Innovations: Upgraded Kafka infrastructure with centralized client wrappers, migrated real-time pipelines to fStream, and integrated Pulsar with Spark and Storm, scaling event throughput from 10K to 22K events/sec
  • Leadership & Mentorship: Authored comprehensive design documents, contributed to non-functional requirement (NFR) planning and scalability testing for major sales events, and actively mentored junior colleagues through knowledge-sharing sessions
Technologies:
Apache Spark Hadoop Kafka Pulsar Spark SQL Parquet HDFS Storm fStream

Software Engineer

Fiorano Software Technologies Pvt. Ltd.

Bengaluru, India

Jul 2017 - Oct 2018

Middleware product-based company

  • Product Enhancement: Enhanced Fiorano's Eclipse-based eStudio tool by integrating ESB, Adaptors, and Fiorano MQ, driving a seamless middleware experience
  • Core Development: Utilized Java and the Eclipse Modelling Framework (EMF) for effective data modelling and developed key features for importing/exporting global funclets
  • Localization & Quality: Localized eStudio for the Japanese market, improved UI components for profile validation and debugging, and resolved 70+ bugs to ensure robust performance
Technologies:
Java Eclipse EMF ESB Fiorano MQ

Core Skills & Technologies

Programming Languages

  • Java
  • Python
  • Scala
  • C++
  • C
  • GoLang

Data Engineering & Big Data

  • Apache Spark
  • Apache Hadoop
  • Apache Kafka
  • Apache Pulsar
  • Apache Airflow
  • Databricks

Cloud & DevOps

  • Google Cloud Platform (GCP)
  • Azure Cloud
  • Azure OpenAI Studio
  • Docker
  • Kubernetes

Databases & Storage

  • Aerospike
  • HBase
  • HDFS
  • Parquet

Web & API Development

  • Flask
  • REST APIs
  • Langchain

Tools & IDEs

  • IntelliJ IDEA
  • PyCharm
  • Maven
  • GitHub Copilot
  • Linux/Mac OS

Awards & Recognition

  • Hackathon Runner-up - Flipkart (Jun 2019)
  • Instant Karma Award - Flipkart (Feb 2019)

Certifications

  • CodeChef Certified Data Structures and Algorithms Programme (Foundation Level)

Languages

  • English (Professional)
  • Hindi (Native)