{"id":2975300,"url":"https://alion.io/job/digirecruitx-pyspark-data-engineer","title":"PySpark Data Engineer","company":{"id":2432154,"name":"DigiRecruitx","domain":"digirecruitx.com","url":"https://alion.io/company/digirecruitx","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"WP Job Openings","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":15500,"max_usd":39000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":21},"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Airflow","optional":false},{"name":"Amazon Kinesis","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Amazon S3","optional":false},{"name":"Anthropic","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"Bitbucket","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude","optional":false},{"name":"Claude Code","optional":false},{"name":"Copilot","optional":false},{"name":"Cursor","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"DynamoDB","optional":false},{"name":"ETL/ELT","optional":false},{"name":"LLM","optional":false},{"name":"OpenAI","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Scrum","optional":false},{"name":"Snowflake","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-07-17T10:08:44Z","employer_posted_date":"2026-07-17","last_verified_at":"2026-10-22T03:02:47Z","board_verified":true,"closed_at":null,"days_open":96,"trust":{"level":"ghost","repost_count":0,"flags":["stale","company_stale"],"days_open":95},"description":"We are looking for a Python / PySpark Data Engineer to design, build, and optimize large-scale data pipelines and analytics solutions on cloud infrastructure. This role is pivotal in enabling data-driven decisions across our portfolio of SaaS businesses. The ideal candidate will partner closely with product, analytics, and engineering teams to deliver reliable, high-performance data platforms.\nResponsibilities\nDesign and develop scalable data pipelines using Python and PySpark on AWS (EMR, Glue, or Databricks).\nBuild and maintain ETL/ELT workflows to process large volumes of structured and semi-structured data across multiple data sources.\nCollaborate with product managers and data analysts to translate business requirements into robust data models and analytical pipelines.\nDesign efficient data schemas and manage data persistence using SQL/NoSQL stores including Redshift, Athena, S3, and DynamoDB.\nImplement data quality frameworks, monitoring, and alerting to ensure pipeline reliability and data integrity.\nLead CI/CD integration for data pipelines using Bitbucket Pipelines or equivalent DevOps tooling for continuous delivery and deployment.\nPromote engineering best practices including pair programming, code reviews, unit testing, and documentation.\nOperate within Scrum teams, contributing to agile ceremonies and driving data engineering domain best practices.\nChampion an AI-first development approach by integrating AI tools and LLM-powered capabilities into data engineering workflows and pipeline automation.\nLeverage Claude Code and Anthropic’s Claude API to accelerate development velocity, automate repetitive data tasks, perform AI-assisted code reviews, and generate high-quality transformation logic and test code.\nMentor the team on responsible and effective use of AI pair-programming tools, prompt engineering best practices, and AI-augmented data pipeline development.\nDesired Qualifications\nB.E./B.Tech in Computer Science, Data Engineering, or a related field; Master’s degree preferred.\nMinimum 4-6 years of experience in data engineering with strong hands-on expertise in Python and PySpark.\nProven experience building and optimizing large-scale data pipelines on AWS (EMR, Glue, Databricks, or equivalent).\nStrong proficiency in SQL and experience with columnar/analytical databases such as Redshift, Athena, or Snowflake.\nExperience with streaming data architectures using Apache Kafka, Spark Streaming, or AWS Kinesis.\nFamiliarity with data orchestration tools such as Apache Airflow.\nWorking knowledge of data lake and lakehouse architectures (Delta Lake, Apache Iceberg, or AWS Lake Formation).\nStrong understanding of DevOps principles and experience with CI/CD tools, specifically Bitbucket Pipelines.\nDemonstrated hands-on experience with an AI-first development approach, including using AI coding assistants, LLM APIs, or generative AI tools to accelerate engineering delivery and product innovation.\nProficiency with Claude Code (Anthropic’s agentic CLI coding tool) or equivalent AI-powered development environments (GitHub Copilot, Cursor, etc.) for daily development tasks, code generation, refactoring, and debugging.\nFamiliarity with prompt engineering, RAG (Retrieval-Augmented Generation), and integrating LLM APIs (Anthropic Claude, OpenAI, etc.) into data pipelines or analytical workflows is a strong plus.\nCTC: Decent Hike on Current CTC","description_format":"text","description_chars":3399,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"master","optional":true},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Recruiting & Staffing"],"lifecycle":[{"event":"open","at":"2026-10-21T04:22:46Z"}],"visa":[],"liveness":{"score":8,"band":"cold","label":"Long shot","p_open":1,"p_active":0.287,"p_room":0.28,"age_days":95,"expected_fill_days":22,"reasons":["conf:1","win:tail","crowd:"],"computed_at":"2026-10-21T05:46:15Z"},"pay":null,"html_url":"https://alion.io/job/digirecruitx-pyspark-data-engineer","json_url":"https://alion.io/job/digirecruitx-pyspark-data-engineer.json","meta":{"generated_at":"2026-10-22T03:31:43Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3331,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-23T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2432154},"rest":"https://alion.io/mcp/rest/get_company?id=2432154"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdigirecruitx-pyspark-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdigirecruitx-pyspark-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdigirecruitx-pyspark-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/digirecruitx-pyspark-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdigirecruitx-pyspark-data-engineer"}]}