{"id":2114045,"url":"https://alion.io/job/codilime-data-engineer-2","title":"Data Engineer","company":{"id":1985033,"name":"CodiLime","domain":"codilime.com","url":"https://alion.io/company/codilime","size_band":"201-500","is_staffing_agency":false,"employer_type":"staffing","is_intermediary":false,"listed_via":null,"ats_vendor":"Teamtailor","truth_index":{"grade":"B","score":80,"open_postings":5,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":9,"computed_at":"2026-10-17T05:46:00Z"}},"role":"Data Science","role_family":"Data Science","seniority":null,"employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"inferred_city","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":["Warsaw, Poland"],"countries":["PL"],"hiring_countries":["PL"],"hiring_countries_total":1,"salary":null,"salary_estimate":{"min_usd":55000,"max_usd":98000,"period":"year","method":"role_country_seniority_unknown","sample_n":361},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"AI Agents","optional":false},{"name":"Airflow","optional":false},{"name":"API Gateway","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"ChatGPT","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude Code","optional":false},{"name":"Cursor","optional":false},{"name":"Databricks","optional":false},{"name":"dbt","optional":false},{"name":"Docker","optional":false},{"name":"ETL/ELT","optional":false},{"name":"FastAPI","optional":false},{"name":"Git","optional":false},{"name":"GitHub Actions","optional":false},{"name":"Kubernetes","optional":false},{"name":"LangChain","optional":false},{"name":"Mypy","optional":false},{"name":"OpenAI","optional":false},{"name":"OpenSearch","optional":false},{"name":"Poetry","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Pydantic","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"Rest API","optional":false},{"name":"Snowflake","optional":false},{"name":"SonarQube","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Azure AKS","optional":true},{"name":"Django","optional":true},{"name":"DuckDB","optional":true},{"name":"Flask","optional":true},{"name":"Go","optional":true},{"name":"Polars","optional":true},{"name":"pySpark","optional":true},{"name":"Redis","optional":true},{"name":"Rust","optional":true},{"name":"Scala","optional":true},{"name":"TypeScript","optional":true}],"status":"live","first_seen_at":"2026-09-04T15:10:08Z","employer_posted_date":"2026-09-04","last_verified_at":"2026-10-17T09:17:41Z","board_verified":true,"closed_at":null,"days_open":42,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":42},"description":"Get to know us better\nCodiLime is a software and network engineering industry expert and the first-choice service partner for top global networking hardware providers, software providers and telecoms. We create proofs-of-concept, help our clients build new products, nurture existing ones and provide services in production environments. Our clients include both tech startups and big players in various industries and geographic locations (US, Japan, Israel, Europe).\nWhile no longer a startup - we have 250+ people on board and have been operating since 2011 we’ve kept our people-oriented culture. Our values are simple:\nAct to deliver.\n\nDisrupt to grow.\n\nTeam up to win.\n\nThe project and the team\nYou will join the team behind a large-scale, centralized data platform built for a global consulting organization. The platform is the shared source of company data behind several of the firm's internal products, and is used daily by consultants for company research.\nThis is fundamentally a data engineering role combined with software engineering: you'll be designing, coding, testing, and operating production Python systems - pipelines, libraries, and services - that move, transform, and serve data at scale. This is not a role focused on configuring tools or writing one-off queries. You'll be building reliable, maintainable software that powers our data platform. The goal is a unified, enterprise-grade dataset of 300M+ company records, integrated from 10+ external and internal sources.\nThe platform delivers firm-level and site-level data - firmographics, technographics, and hierarchical relationships (parent company, subsidiary, site) - alongside key business metrics such as revenue, CAGR, EBITDA, headcount, M&A activity, competitors, industry classification, and web traffic. Data needs to stay accurate, well-structured, and fast to query as both the dataset and the number of consumers keep growing.\nTechnology stack:\nLanguages: Python, SQL\n\nData platform: Snowflake, dbt\n\nWorkflow orchestration: Apache Airflow (complex DAGs), running on Kubernetes\n\nData processing: Apache Spark on Azure Databricks\n\nData tooling and DBs: pandas, Polars, PyArrow, DuckDB, PySpark, PostgreSQL, Redis\n\nCloud: Azure (AKS, Blob Storage, ACR, Databricks, OpenSearch, Azure AI Search)\n\nAPI & services: FastAPI (REST, async), API Gateway\n\nTesting & code quality: pytest, mypy/pyright, ruff/black, sqlfluff, SonarQube\n\nSchema validation: Pydantic\n\nDependency & environment management: uv, Poetry\n\nCI/CD & infrastructure: GitHub Actions, Docker, Kubernetes\n\nAI-Assisted Development: Cursor, Claude Code, ChatGPT Enterprise\n\nFuture direction: agentic AI systems, LangChain, Azure OpenAI integration\n\nWhat else you should know:\nTeam: Data Architecture Lead, Data Engineers, DataOps Engineers, Backend Engineer, Product Owner, collaboration with Frontend Engineers and Data Science and AI Engineers\n\nDistributed team across Europe and India\n\nAgile, collaborative environment; given the platform's organization-wide impact, we're looking for a mature, proactive, results-driven approach\n\nCode quality is enforced through testing, typing, and tooling - not just code review\n\nWe work on multiple interesting projects at a time, so it may happen that we’ll invite you to an interview for another project if we see that your competencies and profile are well suited for it.\nYour role\nThis is a results-driven contributor role, split roughly between hands-on engineering delivery and continuous improvement of the platform.\nAs a part of the project team, you will be responsible for:\nEngineering & delivery (~70%)\n\nDesign, build, and maintain batch and streaming data pipelines in Python, including their orchestration, scheduling, and monitoring (Airflow)\n\nDevelop data transformation logic in SQL and dbt on Snowflake, and build data models that support fast, reliable querying\n\nWrite reusable, well-typed Python libraries and internal packages used by other engineers and analysts\n\nBuild Python services and APIs (FastAPI) that expose data to downstream applications, and integrate with third-party and internal APIs\n\nWrite unit, integration, and data-contract tests, and keep pipelines covered by automated CI\n\nProfile and optimize Python code and data processing jobs for runtime, memory, and cost\n\nDeploy and operate code in the cloud using containers, infrastructure as code, and CI/CD\n\nEnforce access controls, secrets handling, and sensitive-data protections throughout the data lifecycle\n\nUse AI coding assistants effectively while validating all generated output before it reaches production, and help build automated quality gates\n\nContinuous improvement (~30%)\n\nFind and fix efficiency, reliability, cost, and correctness issues in existing pipelines, refactoring toward simpler designs\n\nReplace one-off scripts and notebooks with tested, packaged, scheduled code\n\nImplement data quality checks, validation, and monitoring that catch issues before consumers do\n\nCreate matching logic to deduplicate and connect entities across multiple data sources\n\nDocument data processes and system architecture, and maintain project documentation\n\nDo we have a match?\nAs a Data Engineer, you must meet the following criteria:\nStrong Python experience: data structures, typing, error handling, generators/iterators, context managers, and the standard library\n\nSoftware engineering fundamentals: modular design, dependency management, packaging, and API design\n\nHands-on experience building and operating ETL/ELT pipelines in production, not just scripts or notebooks\n\nStrong experience with Snowflake and dbt\n\nExperience with Apache Airflow or similar code-based orchestration tools\n\nExperience with Apache Spark, ideally on Databricks\n\nSolid SQL and data modelling, sufficient to design robust database schemas and query them effectively, including PostgreSQL\n\nTesting discipline across both code and data: pytest (fixtures, mocking, code that's testable by construction) and dbt data tests, with data-contract checks in CI\n\nExperience with Docker, Kubernetes, and CI/CD practices\n\nDebugging and profiling skills - able to reason about performance, concurrency, and memory in Python\n\nExperience with at least one public cloud (AWS or Azure)\n\nExperience with version control systems (Git)\n\nExperience using AI coding assistants such as Claude Code, Cursor, or similar on a daily basis\n\nStrong communication skills and good knowledge of English (minimum C1 level).\n\nBeyond the criteria above, we would appreciate the following nice-to-haves:\nExperience with Pydantic or similar schema-validation libraries\n\nPython web/API frameworks (FastAPI, Flask, Django)\n\nAsync Python, multiprocessing, or other concurrency patterns\n\nExperience with Azure AI Search or AWS OpenSearch\n\nA second language: Go, Rust, Scala, or TypeScript\n\nFamiliarity with LLMs, Azure OpenAI, or agentic AI systems\n\nMore reasons to join us\nFlexible working hours and approach to work: fully remotely, in the office or hybrid\n\nProfessional growth supported by internal training sessions and a training budget\n\nSolid onboarding with a hands-on approach to give you an easy start\n\nA great atmosphere among professionals who are passionate about their work\n\nThe ability to change the project you work on","description_format":"text","description_chars":7251,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"Advanced (C1)","optional":false}]},"benefits":["Flexible schedule"],"hiring_locations":[{"name":"Poland","iso":"PL","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Cybersecurity","Design & Creative"],"lifecycle":[{"event":"open","at":"2026-10-08T23:15:21Z"}],"visa":[],"liveness":{"score":9,"band":"cold","label":"Long shot","p_open":1,"p_active":0.263,"p_room":0.35,"age_days":42,"expected_fill_days":9,"reasons":["conf:9","velocity","win:tail"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/codilime-data-engineer-2","json_url":"https://alion.io/job/codilime-data-engineer-2.json","meta":{"generated_at":"2026-10-17T11:25:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":14876,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1985033},"rest":"https://alion.io/mcp/rest/get_company?id=1985033"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcodilime-data-engineer-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcodilime-data-engineer-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcodilime-data-engineer-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/codilime-data-engineer-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcodilime-data-engineer-2"}]}