Rakesh Pandey
Data Scientist
8years leading teams
5years in tech
16years of work
5repositories checked by code analysis
Overview
A Data Engineer with practical experience building Databricks lakehouse ingestion and a small production-style API adapter for weather data; the strongest proven skill is building resilient ETL and ingestion logic as shown by Autoloader bronze ingestion and Spark transformation scripts; there is little or no evidence of statistical modeling, formal test coverage, or MLOps for predictive models.
Phone
Timeline
Solution Manager (Data Architecture)
·
Lead
Amdocs Inc.
·
Full-Time
Owned end-to-end, customer-facing data architecture and delivery across DI, EDW, reporting and campaign operations, from source feasibility to SQL/PySpark validation and production issue resolution. Served as the U.S. solution owner for migration eligibility and reconciliation across DI/ODS/Hive/EDW layers. Led an offshore engineering/testing/DBA team and coordinated broader ITO stakeholders to manage data defects, release dependencies, SLA recovery, and production continuity.
Databricks
pySpark
SQL
Development Manager (Data Architecture)
·
Lead
Amdocs India
·
Full-Time
Led architecture, development and production support for ODS/data platform processing for multiple Telefónica affiliates, covering integration changes, UAT and production handoff. Improved batch throughput and enabled more frequent near-real-time micro-batch execution using Databricks/PySpark profiling and SQL/BTEQ tuning. Designed a rule-based PL/SQL purge and archival framework to reduce storage while preserving data needed for operations and downstream reporting, and drove resolution of recurring data-quality and integration issues.
Databricks
pySpark
SQL
DWH/BI Team Lead (Data Architecture)
·
Lead
Amdocs India
·
Full-Time
Led data architecture and engineering workstreams across Telefonica and Three Ireland, including solution analysis, ETL/data validation, UAT strategy, migration readiness and production support. Owned migration and cutover planning for ODS/data platform transitions, including validation and reconciliation activities and transition back to BAU. Defined UAT readiness and sign-off criteria for data-intensive releases and worked with customer and third-party teams to establish representative test data for complex cross-system scenarios.
Senior SME / SME
·
Senior
Amdocs Development Center India
·
Full-Time
Built and supported DWH/BI integrations using Informatica, Unix/Shell and PL/SQL, covering source-to-target analysis, ETL troubleshooting and data-quality resolution. Developed components of a Unix-based ingestion approach for extracting data from Oracle and loading into Oracle and Netezza targets using script-driven processing and scheduling. Completed customer-facing onsite delivery and knowledge transfer assignments related to migration, data validation and sign-off.
Informatica
Unix
Systems Engineer
·
Middle
Tata Consultancy Services
·
Full-Time
Delivered ETL/BI testing and data validation for a banking client and supported reliable release delivery through reusable test automation. Built VBA-based utilities to streamline validation efforts and improve testing consistency during project work. Focused on ensuring data correctness for downstream reporting and integration scenarios.
Projects
Created a Databricks-hosted service that ingests and normalizes National Weather Service forecasts and alerts. The solution generates embeddings with Sentence Transformers and stores vectors in a PostgreSQL/Lakebase-style storage using pgvector. Exposes Flask APIs for cosine-similarity search, filtering, and optional indexing to improve retrieval performance.
Built a Databricks-based assistant for pipeline reliability investigations that normalizes heterogeneous infrastructure logs with PySpark. It retrieves relevant runbooks and incident knowledge using vector search and supports investigation workflows via a Flask application. Uses a Lakebase/PostgreSQL-style storage approach with pgvector similarity search to surface actionable context.
Technical skills
Languages
2
SQL
Python
Python
3
pySpark
Requests
Flask
Databases
3
Databricks
PostgreSQL
Delta Lake
DevOps
3
Unix
AWS
Azure
Other
25
Informatica
LLM
Spark
Google BigQuery
Oracle
Sentence-Transformers
Transformers
pgvector
Teradata
GitHub
Gunicorn
Tableau
Jupyter Notebook
Azure Data Factory
Cognos
Copilot
RAG
DataStage
Talend
Embeddings
MLOps
Prompt Engineering
Dimensional Modeling
ETL/ELT
Semantic Search
Middle AI/ML Engineer
Confidence: Medium LLM Engineer
LLM engineer (mid-level) who builds agent-facing MCP tools and retrieval-oriented ingestion pipelines. The strongest proven skill is implementing agent tool endpoints and a retrieval pipeline that encodes and stores runbook and GitHub issue embeddings, evidenced by weather-mcp-agent/mcp_server/weather_mcp_server.py and pipeline-reliability-copilot/ingest_knowledge.py plus mcp_app/services.py. There is no public evidence of custom model training, systematic experiment tracking, or advanced performance engineering.
Model Architecture & Training
1/10
How well models are designed and trained
Minimal model architecture or training engineering; the code uses off-the-shelf sentence-transformers for embeddings and contains no custom training loops, fine-tuning logic, or architecture design.
Evidence
pipeline-reliability-copilot/mcp_app/services.py:get_embedder and embed_texts - uses SentenceTransformer(MODEL_NAME)
pipeline-reliability-copilot/ingest_knowledge.py:store_embeddings - encodes chunks and stores vector literals
Data Pipeline & Feature Engineering
5/10
How data is prepared for models
Clear, pragmatic data engineering and feature extraction work: a Databricks-targeted Spark ETL normalizes heterogeneous logs, a Python ingestion pipeline builds documents and stores embeddings, and code provides chunking/embedding logic for semantic search.
Evidence
pipeline-reliability-copilot/spark/ingest_pipeline_logs.py:load_raw_rows, parse_fields, build_records, run_spark - Spark ETL and parsing logic
pipeline-reliability-copilot/ingest_knowledge.py:ingest_runbooks and ingest_github - upsert_document and store_embeddings
pipeline-reliability-copilot/mcp_app/services.py:chunk_text and load_curated_runs_into_db - chunking and DB load logic
Experimentation & Evaluation
2/10
How results are measured and tested
Light experimentation and validation tooling: notebooks and scripts validate database schema and ingest flow, but there is no experiment tracking, systematic evaluation pipelines, or ablation/benchmark code.
Evidence
pipeline-reliability-copilot/2_Lakebase validation.ipynb - notebook validating schema, counts and running ingestion
pipeline-reliability-copilot/ingest_knowledge.py:main - prints counts and flow logging for ingestion
MLOps & Deployment
4/10
How models are shipped to production
Practical MLOps and deployment integration: code exposes MCP tools, a small Flask/Databricks app surface, parameterized DB upserts, and instructions for deploying an MCP server; good separation of service and transport layers is present.
Evidence
weather-mcp-agent/mcp_server/weather_mcp_server.py:@mcp.tool handlers and mcp.run for deploying the MCP server
pipeline-reliability-copilot/mcp_app/run.py and mcp_app/mcp_server.py - MCP app wiring and FastMCP usage
pipeline-reliability-copilot/ingest_knowledge.py and lakebase.py - DB upsert patterns and schema interactions
Computational Efficiency
2/10
How efficiently computing resources are used
Minor efficiency considerations are present (cached embedder, batched model.encode) but there are no GPU/quantization, memory profiling, or large-scale performance optimizations.
Evidence
pipeline-reliability-copilot/mcp_app/services.py:get_embedder uses lru_cache to reuse the model instance
pipeline-reliability-copilot/mcp_app/services.py:embed_texts calls model.encode on lists of texts (batched encoding)
Research Depth & Innovation
1/10
Depth of research and new ideas
No research-level contributions or custom algorithm development; the code reuses established libraries and models without paper re-implementation or novel layers.
Evidence
pipeline-reliability-copilot/mcp_app/services.py: uses 'sentence-transformers/all-MiniLM-L6-v2' without custom model code
No custom training loop or paper-implementation files present
Expertise
AI / LLM Engineering (Agents)· Middle
RAG· Middle
Industries
Artificial Intelligence· Middle
Data & Analytics· Middle
Technologies
MLOps
PostgreSQL
Copilot
Spark
Databricks· since 2021
Oracle
pgvector
Sentence-Transformers
Embeddings
Prompt Engineering
Azure
Transformers
AWS
LLM
RAG
Google BigQuery
Teradata
GitHub
Semantic Search
Unix· since 2012
Recommendations
- Develop and extend MCP/agent tool integrations and adapters for LLMs and orchestration platforms.
- Build retrieval pipelines and production RAG stacks that include embedding ingestion, vector stores, and query routing.
- Implement Databricks-focused MLOps flows: automated ingestion, schema migration, CI/CD for notebooks, and deployment of MCP services.
- Add experiment tracking and evaluation (MLflow/W&B), and pursue production-focused efficiency work (batching, GPU usage, quantization) where latency/cost matters.
Repositories
The developer's experience in this domain has been verified based on AI analysis of the following repositories:
Middle Data Scientist
Confidence: Medium Data Engineer
A Data Engineer with practical experience building Databricks lakehouse ingestion and a small production-style API adapter for weather data; the strongest proven skill is building resilient ETL and ingestion logic as shown by Autoloader bronze ingestion and Spark transformation scripts; there is little or no evidence of statistical modeling, formal test coverage, or MLOps for predictive models.
Statistical Rigor
Correct use of statistics
Not evidenced in public code
Data Wrangling & Cleaning
5/10
Preparing and cleaning data
Clear production-focused ETL and ingestion patterns, schema enforcement, streaming tables, SCD handling, and defensive API handling for external data sources.
Evidence
azure-databricks-lakehouse-poc/customer_order_pipeline/transformations/01_bronze_ingestion.sql: use of cloud_files Autoloader for bronze ingestion and ingest timestamping
azure-databricks-lakehouse-dab/feature_dab_job_pipeline_project/src/python_script/order_silver.py: spark table transformations and data validation filter for order_silver
weather-mcp-agent/mcp_server/weather_broker.py: _get_session, _request_json with timeout handling and custom exceptions
Exploratory Analysis & Visualization
Exploring and visualizing data
Not evidenced in public code
Predictive Modeling
Building models that predict
Not evidenced in public code
Business Insight & Impact
2/10
Turning analysis into business value
Some business-facing artifacts (materialized views and customer snapshot) connect data to simple business metrics but lack deeper cost/impact analysis or decision framing.
Evidence
azure-databricks-lakehouse-dab/feature_dab_job_pipeline_project/src/sales_pipeline/transformations/gold_transform.sql: customer_snapshot and customer_order_payment_gold views for business metrics
Reproducibility & Notebook Hygiene
3/10
Clean, repeatable analysis
Notebooks and job scripts show some operational hygiene and parameterization, but pinned environments, CI, formal data versioning or unit tests are missing.
Evidence
azure-databricks-lakehouse-dab/feature_dab_job_pipeline_project/src/python_script/order_bronze.py: argparse and dbutils widget handling for environment parameterization
azure-databricks-lakehouse-poc/notebooks/AutoLoader.ipynb: notebook contains NOTEBOOK META and documented ingestion steps
Expertise
Big Data· Middle
Data Warehouse· Middle
Streaming· Middle
Technologies
SQL· since 2021 · Middle
Delta Lake
Jupyter Notebook
Recommendations
- Develop production Databricks ETL pipelines and streaming ingestion using Autoloader, Delta Lake, and SCD flows.
- Build lightweight API adapters and microservices for external data sources with robust error handling and retries.
- Implement test suites and CI for data pipelines including unit tests for transformation logic and integration tests for ingestion flows.
- Operationalize data platform jobs with parameterization, secrets management, and documented deployment pipelines.
Repositories
The developer's experience in this domain has been verified based on AI analysis of the following repositories:
Middle Backend Developer
Confidence: Medium Data Platform
Backend Python engineer (Middle level) focused on data engineering and tool integrations with a strength in building ETL and embedding pipelines. The strongest proven skill is normalizing and ingesting operational logs and embedding/document upsert workflows as implemented in spark/ingest_pipeline_logs.py and ingest_knowledge.py. There is limited evidence of DB migration history, systematic observability/metrics, large-scale distributed system design, or extensive automated test coverage.
API Design
4/10
How well APIs are designed
Reasonable API/tooling design with consistent error responses and query limits, but no formal versioning, idempotency strategy, or documented API contract.
Evidence
weather-mcp-agent/mcp_server/weather_mcp_server.py: @mcp.tool handlers get_current_weather/get_forecast/get_travel_recommendation/compare_weather with uniform _error_response
pipeline-reliability-copilot/services.py: list_pipeline_runs(limit) and search_knowledge(query, top_k) showing pagination/limit patterns and search parameters
Data Layer & Database
5/10
Working with databases
Solid data layer practices: explicit DB connection handling, parameterized SQL, upserts and vector storage usage, but no migration history files provided.
Evidence
pipeline-reliability-copilot/lakebase.py: get_connection contextmanager with commit/rollback and parameterized execute/fetch helpers
pipeline-reliability-copilot/services.py: load_curated_runs_into_db INSERT ... ON CONFLICT upsert pattern
pipeline-reliability-copilot/ingest_knowledge.py: store_embeddings uses literal vector insertion and chunking for pgvector-style storage
Scalability & Performance
4/10
Handling load and speed
Performance-aware choices such as HTTP session reuse, request timeouts, cached ML embedder and a Spark-based ETL demonstrate attention to scale, but production tuning and cache invalidation strategies are minimal.
Evidence
weather-mcp-agent/mcp_server/weather_broker.py: _get_session() reuses requests.Session and HTTP_TIMEOUT_SECONDS usage
pipeline-reliability-copilot/services.py: @lru_cache get_embedder to avoid reloading model
pipeline-reliability-copilot/spark/ingest_pipeline_logs.py: DataFrame-only transforms (run_spark) for large-scale ETL on Databricks
System Architecture
4/10
Overall system structure
Clear module boundaries and separation of concerns between broker, server tools, ETL, and DB helpers, but no complex multi-service contracts or advanced degradation strategies.
Evidence
weather-mcp-agent/mcp_server/weather_broker.py and weather_mcp_server.py: separation of HTTP API adapter and MCP tool surface
pipeline-reliability-copilot/lakebase.py and pipeline-reliability-copilot/services.py: isolated DB access layer and business logic
pipeline-reliability-copilot/ingest_knowledge.py and spark/ingest_pipeline_logs.py: ETL and ingestion split into discrete steps
Security & Auth
4/10
Protecting data and access
Good basic security hygiene such as input validation, request timeouts, and parameterized SQL; secret retrieval via Databricks is used, but there is limited evidence of auth/token lifecycle management or dependency auditing.
Evidence
weather-mcp-agent/mcp_server/weather_broker.py: _request_json handles requests.Timeout and RequestException and validates JSON payloads
pipeline-reliability-copilot/lakebase.py: resolve_database_url reads Databricks secrets and falls back to environment variables
pipeline-reliability-copilot/lakebase.py and services.py: parameterized SQL usage in execute/fetch_all avoiding raw string interpolation
Reliability & Observability
3/10
Stability and monitoring
Basic reliability and observability patterns exist such as logging, timeouts, and transactional rollback, but there is little evidence of systematic retry/backoff policies, structured correlation ids, metrics, alerts or extensive test coverage.
Evidence
weather-mcp-agent/mcp_server/weather_mcp_server.py: logging setup and logger.warning in exception handlers
pipeline-reliability-copilot/lakebase.py: get_connection contextmanager with commit/rollback for transactional safety
weather-mcp-agent/mcp_server/test_broker.py: simple smoke tests demonstrating manual verification steps
Expertise
Backend AI & LLM· Middle
Databases & Vector Storage· Middle
Python· Middle
Microservices & API Architecture· Middle
Industries
Artificial Intelligence· Middle
Data & Analytics· Middle
Technologies
Python· Middle
Flask
pySpark· since 2021
Gunicorn
Requests
Recommendations
- Develop an incident-search microservice that exposes the pgvector search with pagination and clear API contracts and versioning.
- Add schema migrations and a documented migration history (e.g. Alembic) and introduce a connection-pooling strategy for DB access.
- Improve observability by adding structured logging with correlation ids, metrics export, and retry/backoff with jitter for external calls.
- Expand automated tests including unit tests for services and integration tests for lakebase and embedding upsert paths.
Repositories
The developer's experience in this domain has been verified based on AI analysis of the following repositories:
Job preferences
- Work permits
- India
Open to AI agents: they read this profile and start a dialog through Alion
alion.io/@rakesh-pandey/mcpAlion · People, companies and AI agents in one graphalion.io/@rakesh-pandey