{"id":2642880,"url":"https://alion.io/job/datapelago-data-processing-engineer-io","title":"Data Processing Engineer - I/O","company":{"id":4422787,"name":"DataPelago","domain":"datapelago.ai","url":"https://alion.io/company/datapelago","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"BambooHR","truth_index":{"grade":"D","score":40,"open_postings":12,"ghost_share":1,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-17T05:46:00Z"}},"role":"Data Science","role_family":"Data Science","seniority":"junior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"Apache Iceberg","optional":false},{"name":"C++","optional":false},{"name":"CI/CD","optional":false},{"name":"Flink","optional":false},{"name":"Linux","optional":false},{"name":"Machine Learning","optional":false},{"name":"Ray","optional":false},{"name":"Rust","optional":false},{"name":"Spark","optional":false},{"name":"Trino","optional":false}],"status":"live","first_seen_at":"2025-04-02T00:00:00Z","employer_posted_date":"2025-04-02","last_verified_at":"2026-10-17T19:31:39Z","board_verified":true,"closed_at":null,"days_open":563,"trust":{"level":"ghost","repost_count":0,"flags":["stale","company_stale"],"days_open":563},"description":"Data Processing Engineer - I/O\nMountain View, CA / Hyderabad, IN / Remote\nAbout DataPelago:\nDataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing.\nThe Opportunity:\nAs a Data Processing Engineer - I/O, you will be a key individual contributor in advancing data\nread and write capabilities of DataPelago’s data processing engine. You will enhance functional\nbreadth, performance, scale, and reliability of the DataPelago engine in reading and writing large scale data of various data types from diverse data sources and data sinks. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.\nWhat You'll Do:\nArchitect: Influence the architecture of how our data processing engine interfaces with data\nsources and sinks, catalogs, data formats.\nDesign: Lead design of functional and performance enhancements to adapters/connectors,\ndata representations, data filtering, caching and more in our data processing engine.\nCore Development: Individually design, implement, test, optimize, and maintain components of the data processing engine.\nInnovation and Differentiation: Analyze technology roadmap of existing and emerging data\nformats and libraries, open table formats, catalog services, and more (e.g., Apache Arrow,\nApache Parquet, Apache Iceberg) and identify opportunities for our engine to enhance technology and product leadership.\nCollaboration: Partner effectively with engineering and product management in defining and\nrealizing the data I/O roadmap of our product..\nContinuous Improvement: Foster best practices in design and code reviews, testing, CI/CD,\nand issue resolution to maintain highest product quality, security, efficiency, & productivity.\nWhat You'll Bring:\nBachelor's degree in Computer Science or a related field with 7+ years of relevant experience OR a Master's degree in Computer Science or a related field with 5+ years of relevant\nexperience.\n3+ years of deep technical experience in developing and optimizing data read and write interfaces for large-scale data processing, particularly related to Apache Parquet, Apache\nORC, Apache Iceberg, Apache Spark, and similar technologies.\nDemonstrated experience in instrumenting, analyzing, and optimizing the performance of\ndata processing engine components on benchmark and customer workloads.\nDemonstrated experience in the design, development, and successful release of high-performance data processing engine features for large production deployments.\nGood knowledge of the architecture of one or more of Apache Spark, Apache Flink, Presto/\nTrino.\nExceptional programming skills in C, C++. Rust experience preferred.\nExtensive development experience in Linux environments.\nStrong analytical and problem-solving skills with a passion for performance optimization.\nLocation Considerations:\nWe value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations.\nWhy Join DataPelago?\nTechnology Leadership: Shape the architecture and development of how our core engine\nworks with advanced data store platforms.\nCutting-Edge Innovation: Work on challenging problems at the forefront of accelerated\ncomputing and data processing.\nSignificant Impact: Your contributions will directly impact the performance and scalability\nof our mission-critical platform.\nGrowth: Expand your technical expertise and scope of responsibilities working with other\ntalented engineers and with a growing product.\nCompetitive compensation, stock options, comprehensive benefits package, leadership de-\nvelopment opportunities.","description_format":"text","description_chars":4260,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Stock options"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-15T19:23:25Z"}],"visa":[],"liveness":{"score":2,"band":"cold","label":"Long shot","p_open":1,"p_active":0.062,"p_room":0.28,"age_days":563,"expected_fill_days":19,"reasons":["conf:3","stale_co","ghost","win:tail","crowd:junior"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/datapelago-data-processing-engineer-io","json_url":"https://alion.io/job/datapelago-data-processing-engineer-io.json","meta":{"generated_at":"2026-10-17T20:26:04Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":4258,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4422787},"rest":"https://alion.io/mcp/rest/get_company?id=4422787"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdatapelago-data-processing-engineer-io"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdatapelago-data-processing-engineer-io"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdatapelago-data-processing-engineer-io"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/datapelago-data-processing-engineer-io\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdatapelago-data-processing-engineer-io"}]}