Senior Data Engineer

POSITION - Data Engineering

LOCATION- Bangalore/Hyderabad/Pune/Coimbatore

EXPERIENCE- 5 to 8 Years

ABOUT HASHEDIN

We are software engineers who solve business problems with a Product Mindset for leading global organizations. By combining engineering talent with business insight, we build software and products that can create new enterprise value.

The secret to our success is a fast-paced learning environment, an extreme ownership spirit, and a fun culture.

WHY SHOULD YOU JOIN US

With the agility of a start-up and the opportunities of an enterprise, every day at HashedIn, your work will make an impact that matters.

So, if you are a problem solver looking to thrive in a dynamic fun culture of inclusion, collaboration, and high performance – HashedIn is the place to be!

From learning to leadership, this is your chance to take your software engineering career to the next level.

So, what impact will you make

Visit us @ https://hashedin.com

JOB TITLE:Data Engineering

OVERVIEW OF THE ROLE:

As a Tech Lead – Data Engineering, you will lead the design of, and stay hands-on in building and optimizing, secure, scalable data pipelines and lakehouse/warehouse platforms on Databricks or Snowflake that power business-critical analytics, data science, and AI applications. You will play a central role in setting technical direction and in implementing and automating batch and, where required, streaming data workflows using modern cloud technologies (AWS, GCP, or Azure). You will guide a team of data engineers, set engineering standards, and own technical delivery and quality. Working within cross-functional teams and with client stakeholders, you will deliver well-engineered, high-quality code and data models, enable AI-ready data for RAG, LLM, and ML use cases, build and deploy AI agents using Agent Bricks and Snowflake Cortex, and drive best practices for data reliability, lineage, quality, governance, cost efficiency, and security.

Mandatory Skills:

• Hands-on software coding in Python, Scala, or Java, along with strong SQL, for building data pipelines, for minimum 5 years

• Hands-on experience with Databricks or Snowflake for at least 3 years (working knowledge of the other is a plus)

• Stakeholder management experience for at-least 2 years, translating business requirements into data solutions

• Experience leading or mentoring a small team of engineers (design reviews, code reviews, delivery ownership) for at least 1 year

• Experience designing data solutions on at least one cloud platform (AWS, GCP, or Azure) and its native data services

Key Responsibilities:

Technical leadership and delivery

• Lead architecture and technical design for data pipelines and platforms on Databricks or Snowflake, owning design reviews and documenting key decisions and trade-offs.

• Lead a small team of data engineers: guide day-to-day work, allocate tasks, and uphold engineering standards through code and design reviews.

• Own technical delivery: estimate, plan, and break down work, flag and mitigate technical risks, and ensure on-time, high-quality releases in an Agile setup.

• Partner with client and business stakeholders to translate requirements into technical solutions, and communicate designs, trade-offs, and status clearly.

Pipeline design and development

• Design, build, and optimize scalable batch and streaming data pipelines and ETL/ELT workflows on Databricks or Snowflake, using Spark (PySpark/Scala), SQL, and orchestration tools such as Airflow, Prefect, or Databricks Workflows.

• Architect efficient solutions for high-volume, batch and, where required, real-time streaming and event-driven data processing, leveraging best-in-class patterns and frameworks.

• Architect and implement Medallion (Bronze/Silver/Gold) and lakehouse/warehouse architectures using Delta Lake, Delta Live Tables/Lakeflow, Snowpipe, Streams, Tasks, and Dynamic Tables to support analytics, data science, AI, and BI workloads.

• Write performant, production-grade SQL for complex data aggregation, transformation, and analytics tasks, and lead data modeling, schema design, and partitioning/clustering strategies for the team's solutions.

Quality, governance, and security

• Establish and own data quality and consistency through validation, cleansing, anomaly detection, and reconciliation, using frameworks such as Great Expectations or Deequ.

• Implement, document, and extend data governance, lineage, and cataloging using Unity Catalog, Snowflake governance features, or tools such as Purview, Glue, Atlan, Alation, IDMC, etc.

• Apply and enforce data security, privacy, and compliance controls (access control, PII masking, retention policies, GDPR/CCPA) across the data platform.

Operations and optimization

• Define and automate deployment and operations of pipelines using CI/CD, containerization, and infrastructure-as-code (Databricks Asset Bundles, Terraform), with monitoring and observability in place.

• Drive platform performance and cost optimization through cluster and warehouse sizing, file and partition tuning, query profiling, and resource monitoring.

AI enablement and collaboration

• Build governed, AI-ready data pipelines that support RAG, LLM, and ML use cases using Databricks Mosaic AI or Snowflake Cortex, including preparing and chunking unstructured data.

• Design, build, and deploy AI agents using Agent Bricks on Databricks or Cortex Agents (with Cortex Search and Cortex Analyst) on Snowflake, including evaluation, monitoring, and governed access to enterprise data.

• Partner with Data Scientists, Analysts, Product Owners, and DevOps engineers to ingest, structure, and expose structured, semi-structured, and unstructured data for diverse use cases.

Ownership and engineering excellence

• Take ownership of the end-to-end pipeline lifecycle across the team's deliverables (design, development, code review, testing, deployment, monitoring, and troubleshooting), and build and drive adoption of reusable modules and frameworks that improve maintainability.

• Mentor and coach engineers, lead technical discussions, and stay current with emerging data and AI technologies, driving their adoption where valuable.

Skills & Experience:

Core skills

• Proficiency in Python, Scala, or Java, with strong software engineering practices (version control, testing, modularity), and the ability to set coding standards through code reviews, and in SQL for ETL/ELT, analytics, and performance optimization.

• Proficiency with Apache Spark (PySpark or Scala) and data pipeline orchestration using Airflow, Prefect, Luigi, or similar tools, including performance tuning at scale.

• Experience with cloud data ecosystems (AWS, GCP, or Azure) and cloud-native data processing services (Glue, Dataflow, Dataproc, EMR, Synapse, etc.), with the ability to design secure, cost-efficient cloud data architectures.

• Strong knowledge of data modeling (star/snowflake, normalized, denormalized) and metadata/catalog management.

Platform expertise

• Databricks: Hands-on experience with PySpark/Spark SQL, Workflows, cluster tuning, and Photon. Skilled in Delta Lake and lakehouse architecture (MERGE/CDC, OPTIMIZE/Z-ORDER, time travel), and in building Medallion (Bronze/Silver/Gold) pipelines using Delta Live Tables/Lakeflow, Auto Loader, and Structured Streaming.

• Snowflake: Alternatively, hands-on experience with Snowflake architecture (virtual warehouses, micro-partitions, clustering) and ELT using Snowpipe/Snowpipe Streaming, Streams, Tasks, Dynamic Tables, and Snowpark (Python), with query profiling and performance tuning.

Quality, governance, and operations

• Experience with data quality frameworks (Great Expectations, Deequ, or custom validation), monitoring/observability tools, and automated testing.

• Hands-on knowledge of data governance, security, and cost optimization, including access control and data protection using Unity Catalog (lineage, RBAC, row/column security) and Snowflake (masking/row access policies, tagging, secure sharing), and cost control through warehouse sizing, resource monitors, and cluster/file optimization; understanding of GDPR/CCPA/HIPAA requirements.

• Hands-on experience with DevOps and infrastructure-as-code for data platforms: CI/CD, Databricks Asset Bundles, Terraform, Docker, and Kubernetes.

AI/GenAI

• Experience building governed data pipelines for RAG, LLM, and ML use cases using Databricks Mosaic AI (Model Serving, Vector Search, MLflow, AI Functions) or Snowflake Cortex (LLM functions, Cortex Search/Analyst, Document AI, Snowpark ML), including preparing and chunking unstructured data.

• Hands-on experience building and deploying AI agents with Agent Bricks (Databricks) or Cortex Agents (Snowflake), including grounding on enterprise data, agent evaluation and monitoring, and governed data access.

Technical leadership

• Experience leading design and code reviews, estimating and planning work, and guiding a small team of engineers in an Agile environment.

Good to have

• Understanding of streaming architectures (Kafka, Kinesis, Pub/Sub, Flink, Spark Streaming).

• Familiarity with BI tools (Power BI, Tableau, Looker).

• Exposure to open-source data tools (dbt, Apache Iceberg, Great Expectations) and DevOps/MLOps practices.

• Knowledge of ML pipeline integration (feature stores, model serving, lineage/monitoring integration) is beneficial.

• Experience implementing data cataloging, stewardship, and data democratization initiatives.

• Prior work in highly regulated or large-scale enterprise data environments (finance, healthcare, or similar).

• Previous experience with data migrations, modernization, or refactoring legacy ETL processes to modern cloud architectures is a strong plus.

Professional Attributes:

• Strong analytical and problem-solving skills, with attention to detail and a commitment to code quality, testing, and documentation.

• Clear communicator who can explain technical designs, trade-offs, and issues to engineers and business stakeholders, including client leadership, and translate business requirements into data solutions.

• Self-starter and fast learner with an extreme ownership mindset, who thrives in dynamic, fast-paced, collaborative environments.

• Curious about emerging data and AI technologies (Databricks, Snowflake, GenAI, AI agents).

• Passion for mentoring and growing engineers, sharing knowledge, and raising the technical bar for data engineering practices; pragmatic decision-maker who balances delivery timelines, quality, and cost.

QUALIFICATIONS:

• Bachelor's or Master's degree in Computer Science or a related field (or equivalent practical experience).

• Certifications preferred: Databricks Certified Data Engineer (Associate/Professional), SnowPro Core/Advanced Data Engineer, or a cloud data certification (AWS Data Engineer, GCP Professional Data Engineer, Azure Data Engineer Associate).

• Experience working in an Agile environment, with exposure to Git, CI/CD, Jira, Confluence, and code review practices.