{"id":3879258,"url":"https://alion.io/job/innovya-technologies-data-engineer","title":"Data Engineer","company":{"id":4401393,"name":"Innovya Technologies","domain":"innovyatech.com","url":"https://alion.io/company/innovya-technologies","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Raipur, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"Databricks","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-11-06T17:23:37Z","employer_posted_date":null,"last_verified_at":"2026-11-06T17:23:37Z","board_verified":false,"closed_at":null,"days_open":0,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":0},"description":"Job Title: Python PySpark\n\nExperience: 6+ years \nJob Type: India, Full-Time, Remote\nInnovya Technologies is a fast‑growing software consulting firm specializing in business automation and modern digital solutions. We enable organizations to accelerate value realization from their technology and transformation initiatives through expert, context‑driven product development and high‑quality engineering services.\nInnovya fosters a high‑energy, collaborative environment built on continuous learning, accountability, and disciplined execution. We value individuals who are results‑oriented, growth‑focused, and motivated to make meaningful contributions. If you thrive in a supportive, fast‑paced setting and enjoy solving complex challenges, Innovya offers an ideal place to grow and make an impact.\nAbout the Role:\n We are looking for an experienced PySpark Developer with deep expertise data streaming development and deployment to join our dynamic team. A Python developer who will play a key role in maintaining, enhancing, and modernizing critical high volume and real time data pipelines. The ideal candidate will have a strong background in building scalable, fault-tolerant, and high-performance ELT applications, with a keen interest in AI enabled development.\nYou will work closely with cross-functional teams to design, analyse, build and deploy complex systems, providing technical direction and expertise to ensure the delivery of robust, efficient, and scalable solutions.\n\nKey Responsibilities:\nDesign, develop, and optimize scalable data pipelines using PySpark \nProcess and analyze large datasets in distributed environments \nCollaborate with data engineers, analysts, and stakeholders to deliver data solutions \nWrite efficient, reusable, and reliable PySpark code \nPerform data cleansing, transformation, and validation \nOptimize Spark jobs for performance and cost efficiency \nIntegrate data from various sources (databases, APIs, streaming platforms, etc.) \nTroubleshoot and debug data pipeline issues \nMaintain documentation for data workflows and processes\nRequired Skills:\n6+ years of total experience in Python development, with at least 5 years in Apache Spark (PySpark).\n· Hands-on experience with Python and PySpark in production data engineering environments. \n· Use Spark DataFrames for large-scale data transformation and processing. \n· Experience with complex aggregations, window functions, conditional aggregations, and performance tuning in Spark. \n· Good understanding of Spark execution concepts such as partitions, shuffles, join behavior, and optimization techniques. \n· Experience with API integrations, including pagination, request handling, response transformation, and fault tolerance. \n· Working knowledge of AWS, preferably with services such as S3, Lambda, Glue, EMR, Athena, Step Functions, or similar. \n· Experience handling JSON and schema-based parsing in distributed data pipelines. \n· Familiarity with SQL-based data extraction and transformation. \n· Ability to write production-quality, maintainable, and testable code. \n· Good analytical and problem-solving skills, with the ability to understand complex data flows and business rules. \n· Experience identifying bugs, troubleshooting processing issues, performing root cause analysis, and implementing fixes in distributed systems. \n· Ability to create and maintain unit tests to improve code quality and reduce regressions.\n· Preferred - 2+ years of DataBricks and Data Lake development experience","description_format":"text","description_chars":3503,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Continuous learning"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Data Engineering & Migration Services"],"lifecycle":[{"event":"open","at":"2026-11-06T17:33:15Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":29,"reasons":["seen:0","win:early"],"computed_at":"2026-11-07T05:47:08Z"},"pay":null,"html_url":"https://alion.io/job/innovya-technologies-data-engineer","json_url":"https://alion.io/job/innovya-technologies-data-engineer.json","meta":{"generated_at":"2026-11-07T08:28:51Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":5101,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-08T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4401393},"rest":"https://alion.io/mcp/rest/get_company?id=4401393"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Finnovya-technologies-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Finnovya-technologies-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Finnovya-technologies-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/innovya-technologies-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Finnovya-technologies-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3879258}},{"tool":"search_jobs","arguments":{"company":"innovyatech.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3879258},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}