Ashwani Verma - Big Data Developer

Overview

Data engineer (mid-level) focused on Spark-based ETL and small-scale Delta Lake validation tests. The strongest proven skill is building and validating a local Delta write/read flow using retailflow-lakehouse/scripts/delta_smoke.py which configures Spark, writes a Delta table, and asserts the expected results. There is little evidence of statistical modeling, production-grade pipeline orchestration, streaming ingestion, or business-impact analysis in public code.

Phone

Timeline

Senior Data Engineer · Senior

Best Buy India · Full-Time

Apr 2025 to Present 1 Year 7 Months In office

Led an enterprise Teradata-to-BigQuery migration on GCP, moving legacy BI workloads while maintaining data accuracy, performance, and production continuity. Built a reusable, metadata-driven Python framework using configuration to standardize vendor program onboarding and reduce duplicated business logic. Drove modernization of Labor domain data workloads and designed a master dataset for downstream reporting with standardized metric definitions. Reworked BigQuery models and SQL frameworks to improve query performance and onboarding speed for additional workloads.

Google BigQuery

Python

SQL

Senior Data Engineer (Project: AXA Belgium) · Senior

Accenture · Full-Time

Jun 2022 to Apr 2025 2 Years 10 Months In office

Provided technical direction and mentored a small engineering team across a multi-year Teradata-to-Azure modernization program. Migrated enterprise data warehouse workloads to Azure using Databricks, Synapse Analytics, and Azure Data Factory while preserving continuity of data operations and performance. Reconstructed legacy Teradata transformation logic through reverse engineering to recreate source-to-Data Mart flows. Implemented Data Vault 2.0 modeling by introducing staging and raw/business vault layers to improve scalability, flexibility, and auditability of the enterprise model.

Databricks

Azure Data Factory

pySpark

MS SQL

SQL

Data Vault

Data Engineer (Project: GAP NA) · Middle

Accenture · Full-Time

Nov 2020 to May 2022 1 Year 6 Months In office

Designed and built ingestion and transformation pipelines on Azure Databricks, applying a medallion architecture with validation checks at each stage. Led data mapping to understand source structures and streamline downstream processing. Orchestrated CI/CD pipelines for Azure runbooks to standardize deployment and integration during the migration. Owned pre-UAT validation with the client and prepared documentation and live demos to support sign-off.

Databricks

pySpark

Spark

SQL

CI/CD

ETL/ELT

JSS Academy of Technical Education

Bachelor's Degree · Computer Science

2016–2020 Lucknow, Uttar Pradesh

Technical skills

Languages

2

SQL

Python

Databases

4

Databricks

MS SQL

Google BigQuery

Delta Lake

DevOps

3

CI/CD

Azure

GCP

Analytics

3

Azure Data Factory

ETL/ELT

Data Vault

Other

11

Spark

pySpark

Teradata

Power BI

GitHub

Looker

Airflow

Jupyter Notebook

Databases

Dimensional Modeling

Microsoft Excel

Middle Data Scientist Confidence: Medium Data Engineer

Data engineer (mid-level) focused on Spark-based ETL and small-scale Delta Lake validation tests. The strongest proven skill is building and validating a local Delta write/read flow using retailflow-lakehouse/scripts/delta_smoke.py which configures Spark, writes a Delta table, and asserts the expected results. There is little evidence of statistical modeling, production-grade pipeline orchestration, streaming ingestion, or business-impact analysis in public code.

Statistical Rigor

1/10

Correct use of statistics

Little to no statistical rigor; artifacts are SQL problem solutions and examples without assumption checks, uncertainty quantification, or formal hypothesis testing.

Evidence

GoogleColaboratory/LeetCode_SQL.ipynb: SQL problem solutions and explanatory comments

Data Wrangling & Cleaning

3/10

Preparing and cleaning data

Basic data wrangling is present: DataFrame creation, SQL temp views and simple parsing are used, plus a small Delta write/read smoke test; however there is limited cleaning, provenance tracking, or leakage prevention beyond simple checks.

Evidence

GoogleColaboratory/LeetCode_SQL.ipynb: data parsing and DataFrame creation

retailflow-lakehouse/scripts/delta_smoke.py: Delta write/read smoke test demonstrating write/read and basic assertions

Exploratory Analysis & Visualization

2/10

Exploring and visualizing data

Minimal exploratory analysis and visualization; code uses show() and markdown explanations for SQL exercises, but there are no analytical plots or question-driven data storytelling.

Evidence

GoogleColaboratory/LeetCode_SQL.ipynb: show() outputs and markdown explanations

Predictive Modeling

Building models that predict

Not evidenced in public code

Business Insight & Impact

1/10

Turning analysis into business value

No demonstrated linkage to business metrics or impact analysis; the work focuses on technical SQL exercises without discussion of KPIs, cost of errors, or actionable business recommendations.

Evidence

GoogleColaboratory/LeetCode_SQL.ipynb: problem framing comments (algorithmic/SQL focus rather than business impact)

Reproducibility & Notebook Hygiene

3/10

Clean, repeatable analysis

Some reproducibility hygiene is visible: pinned dependencies in requirements.txt, explicit PYSPARK_PYTHON usage and a try/finally block to stop Spark; missing are formal environment pinning, CI, data/versioning tooling or seeds.

Evidence

retailflow-lakehouse/requirements.txt: pinned pyspark and delta-spark versions

retailflow-lakehouse/scripts/delta_smoke.py: sets PYSPARK_PYTHON and uses try/finally to stop Spark

Expertise

Big Data· Middle

Technologies

Databases

Spark· since 2020

Delta Lake

Jupyter Notebook

ETL/ELT· since 2020

Recommendations

Develop end-to-end batch ETL pipelines with Spark and Delta Lake including CI tests, automated environment setup, and data versioning (use the existing delta_smoke.py as a starting integration test).
Expand reproducibility by adding environment pinning, a lockfile, CI runs of the smoke tests, and simple data versioning (DVC or similar) to make pipeline runs repeatable.
Prototype streaming ingestion and validation using Spark Structured Streaming and Delta Lake change-logs to broaden from batch to streaming Big Data use cases.
Turn SQL notebooks into parameterized, documented training modules with clearer data-storytelling and business-metric framing to demonstrate impact beyond technical correctness.

Repositories

The developer's experience in this domain has been verified based on AI analysis of the following repositories:

retailflow-lakehouse GoogleColaboratory

Junior AI/ML Engineer Confidence: Medium Data-centric

Data-centric engineer (Junior) focusing on PySpark-based data engineering and Delta Lake smoke testing. The strongest proven skill is building and validating a local Spark + Delta smoke test that writes and reads a Delta table, as implemented in retailflow-lakehouse/scripts/delta_smoke.py. There is no evidence of ML model training, experiment tracking, production deployment pipelines, or extensive automated testing in public code.

Model Architecture & Training

How well models are designed and trained

Not evidenced in public code

Data Pipeline & Feature Engineering

4/10

How data is prepared for models

Clear evidence of basic data engineering and ETL work using Spark and Delta; lightweight but correct data read/write and SQL transformations.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: SparkSession builder and Delta write/read smoke test

GoogleColaboratory/LeetCode_SQL.ipynb: Spark SQL data preparation and window-function SQL solutions

Experimentation & Evaluation

1/10

How results are measured and tested

Minimal evaluation: a smoke assertion verifies read/write correctness but no experiment tracking, validation suites, or reproducible experiment artifacts.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: assert on collected rows and PASS print message after Delta read/write

MLOps & Deployment

1/10

How models are shipped to production

Basic local execution and environment configuration for Spark; no evidence of deployment, serving, model/version management or drift monitoring.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: SparkSession.builder configuration and PYSPARK_PYTHON / SPARK_LOCAL_IP environment settings

Computational Efficiency

2/10

How efficiently computing resources are used

Some attention to runtime parameters (local[2], driver memory, shuffle partitions) indicating awareness of parallelism and resource settings but no profiling or optimization artifacts.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: .master('local[2]') and .config('spark.driver.memory','1g') and spark.sql.shuffle.partitions adjustments

Research Depth & Innovation

Depth of research and new ideas

Not evidenced in public code

Industries

Commerce· Junior

Data & Analytics· Junior

Technologies

Python· since 2025 · Junior

SQL· since 2020 · Senior

Airflow

GCP· since 2025

Databricks· since 2020

Azure· since 2025

MS SQL· since 2022

CI/CD· since 2020

pySpark· since 2020

Google BigQuery· since 2025

Teradata

GitHub

Recommendations

Expand the data engineering work into repeatable pipelines with unit tests and CI integration using pytest and GitHub Actions or similar.
Add data validation and quality checks (for example Great Expectations) and automated smoke/integration tests around Delta writes and reads.
Build end-to-end orchestration (Airflow, Dagster, or Prefect) and add logging/observability for runs and failures to make the pipeline production-ready.

Repositories

The developer's experience in this domain has been verified based on AI analysis of the following repositories:

retailflow-lakehouse GoogleColaboratory

Junior Backend Developer Confidence: Medium Data Platform

Junior data engineer focused on local PySpark and Delta Lake workflows with attention to reproducible local runs. The strongest proven skill is writing and verifying a Delta Lake smoke test as implemented in retailflow-lakehouse/scripts/delta_smoke.py which configures Spark, writes a small Delta table, reads it back, and asserts correctness. There is no public evidence of production-grade pipelines, REST APIs, messaging integrations, schema migration history, or CI-driven tests.

API Design

How well APIs are designed

Not evidenced in public code

Data Layer & Database

3/10

Working with databases

Basic Delta Lake and Spark data-layer work is implemented with local write/read verification and explicit checks for the Delta transaction log.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: expected.write.format("delta").mode("errorifexists").save(str(table))

retailflow-lakehouse/scripts/delta_smoke.py: .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")

retailflow-lakehouse/scripts/delta_smoke.py: assert (table / "_delta_log").is_dir()

Scalability & Performance

2/10

Handling load and speed

Small-scale tuning for local runs is present, but there is no evidence of production scaling, caching strategies, or queue-based decoupling.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: .master("local[2]")

retailflow-lakehouse/scripts/delta_smoke.py: .config("spark.sql.shuffle.partitions", "2")

System Architecture

2/10

Overall system structure

Single-script layout shows attention to environment isolation and unique data paths, but no multi-service decomposition or explicit service contracts.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: table = root / "data" / "smoke" / uuid4().hex

retailflow-lakehouse/scripts/delta_smoke.py: builder = SparkSession.builder.appName("RetailFlow-Delta-Smoke") ... (centralized session builder configuration)

Security & Auth

1/10

Protecting data and access

Minimal environment hygiene is present but there is no auth/authz, secrets management, input validation, or dependency audit logic.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: os.environ["PYSPARK_PYTHON"] = sys.executable

retailflow-lakehouse/scripts/delta_smoke.py: os.environ.setdefault("SPARK_LOCAL_IP", "127.0.0.1")

Reliability & Observability

3/10

Stability and monitoring

Basic reliability practices exist: assertions to validate behavior, log level control, and a try/finally that ensures Spark is stopped.

Evidence

retailflow-lakehouse/scripts/delta_smoke.py: spark.sparkContext.setLogLevel("WARN")

retailflow-lakehouse/scripts/delta_smoke.py: try: ... finally: spark.stop()

retailflow-lakehouse/scripts/delta_smoke.py: assert [(r.order_id, r.amount) for r in rows] == [(0, 10), (1, 20), (2, 30)]

Expertise

Databases & Vector Storage· Junior

Industries

Commerce· Junior

Recommendations

Develop end-to-end ingestion pipelines that include unit and integration tests and store migration history rather than single-shot smoke scripts.
Add CI automation and deterministic tests that run the Delta smoke flow in a reproducible environment and capture metrics/logging.
Work on scaling exercises: introduce streaming ingestion or a queue-based consumer (Kafka/RabbitMQ) and document backoff/retry and idempotency strategies.

Repositories

The developer's experience in this domain has been verified based on AI analysis of the following repositories:

retailflow-lakehouse LeetCode

Job preferences

Work permits: India
Citizenship: India
Employment type: Full-Time