{"id":4696913,"url":"https://alion.io/job/celfocus-data-engineer-azuredatabricks","title":"Data Engineer | Azure/Databricks","company":{"id":54052,"name":"Celfocus","domain":"celfocus.com","url":"https://alion.io/company/celfocus","size_band":null,"is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Teamtailor","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Lisbon, Portugal","Porto, Portugal"],"countries":["PT"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache HTTP Server","optional":false},{"name":"Apache Hudi","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Bicep","optional":false},{"name":"Cassandra","optional":false},{"name":"CI/CD","optional":false},{"name":"CloudFormation","optional":false},{"name":"Data Vault","optional":false},{"name":"Databricks","optional":false},{"name":"dbt","optional":false},{"name":"Delta Lake","optional":false},{"name":"Docker","optional":false},{"name":"DynamoDB","optional":false},{"name":"Embeddings","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Flink","optional":false},{"name":"GCP","optional":false},{"name":"Git","optional":false},{"name":"Google BigQuery","optional":false},{"name":"Kubernetes","optional":false},{"name":"Machine Learning","optional":false},{"name":"MySQL","optional":false},{"name":"Platform Engineering","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Scala","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false},{"name":"FinOps","optional":true},{"name":"Master Data Management","optional":true},{"name":"Snowflake","optional":true}],"status":"live","first_seen_at":"2026-11-20T11:07:28Z","employer_posted_date":"2026-11-20","last_verified_at":"2026-11-20T15:01:06Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Make an impact by working in sectors where technology is the enabler and innovation is the key.\nAt Celfocus, we make data actionable through Next-Gen Intelligence - where data, AI and human creativity come together.\nYou’ll work with multidisciplinary teams and partner with leading companies to turn data into decisions, agility and long-term value.\nExplore your way in.\nThe opportunity:\nThe Data Engineer is responsible for designing, building, and operating modern, scalable, and reliable data platforms that enable data-driven decision-making across the organization.\nThis role plays a critical part in transforming data into a strategic asset by ensuring that data is discoverable, trusted, secure, and accessible to business users, analysts, data scientists, and AI/ML applications.\nThe Data Engineer designs and implements robust data architectures, ingestion frameworks, processing pipelines, and storage solutions capable of supporting both traditional analytics and emerging Artificial Intelligence use cases, including Machine Learning and Generative AI.\nThe role requires a strong understanding of cloud-native technologies, distributed systems, data governance principles, and modern software engineering practices.\nMaintaining a balance between technical excellence and business understanding is essential to ensure that data solutions generate measurable value for the organization.\nYour day-to-day:\nData Platform Engineering\nDesign, develop, and maintain scalable, secure, and resilient data platforms in cloud environments.\n\nBuild and optimize batch and real-time data pipelines across multiple data sources and systems.\n\nImplement and maintain modern data architectures such as Data Lakes, Lakehouses, Data Warehouses, and Data Mesh domains where applicable.\n\nDevelop reusable frameworks and standards for data ingestion, transformation, orchestration, and consumption.\n\nData Integration & Processing\nIntegrate structured, semi-structured, and unstructured data from databases, APIs, streaming platforms, SaaS applications, and external sources.\n\nDevelop and maintain ETL/ELT pipelines ensuring performance, scalability, and maintainability.\n\nDesign efficient data models supporting analytical, operational, and AI-driven workloads.\n\nImplement data transformation and enrichment processes using distributed processing frameworks.\n\nData Quality, Governance & Security\nEnsure high levels of data quality, reliability, lineage, and observability.\n\nDefine and implement data validation, monitoring, and alerting mechanisms.\n\nCollaborate with governance teams to ensure compliance with security, privacy, and regulatory requirements.\n\nApply best practices in access management, encryption, data masking, and information protection.\n\nAI & Advanced Analytics Enablement\nBuild and maintain data foundations that support Machine Learning, predictive analytics, and Generative AI solutions.\n\nPrepare and optimize datasets for ML feature engineering and AI model consumption.\n\nSupport implementation of vector databases, embeddings pipelines, Retrieval Augmented Generation (RAG) architectures, and AI-ready data ecosystems when required.\n\nCollaborate with Data Scientists, AI Engineers, and business stakeholders to operationalize AI use cases.\n\nPlatform Reliability & Operations\nMonitor platform performance, reliability, and cost efficiency.\n\nTroubleshoot and resolve data pipeline failures and platform issues.\n\nOptimize data workloads for scalability, availability, and cloud resource consumption.\n\nPromote DataOps practices through automation, testing, CI/CD, and Infrastructure as Code.\n\nCollaboration & Innovation\nWork closely with Product Owners, Business Analysts, Architects, Data Scientists, and other engineering teams.\n\nContribute to the evolution of data architecture standards and best practices.\n\nStay current with emerging technologies and assess opportunities to enhance the organization's data capabilities.\n\nActively contribute to building a data-driven culture across the organization.\n\nWhat you'll bring:\nCore Data Engineering\nStrong proficiency in SQL and data modelling techniques (dimensional, normalized, Data Vault, etc.).\n\nStrong programming skills in Python and/or Scala.\n\nExperience designing and building large-scale data pipelines.\n\nExperience with batch and streaming data processing architectures.\n\nCloud & Modern Data Platforms\nExperience in at least one major cloud platform:\nMicrosoft Azure (mandatory)\n\nAWS\n\nGoogle Cloud Platform (GCP)\n\nExamples of relevant technologies:\nAzure: Data Factory, Data Lake, Synapse Analytics, Databricks, Fabric\n\nAWS: S3, Glue, EMR, Lambda, Athena, Redshift, Lake Formation\n\nGCP: BigQuery, Dataflow, Dataproc, Pub/Sub\n\nData Processing & Streaming\nExperience with some of the following:\nApache Spark\n\nDatabricks (mandatory)\n\nApache Kafka\n\nApache Flink\n\nApache Beam\n\nAirflow\n\ndbt\n\nDelta Lake\n\nApache Iceberg\n\nApache Hudi\n\nData Storage\nRelational Databases (PostgreSQL, SQL Server, MySQL, Oracle)\n\nNoSQL Databases (MongoDB, Cassandra, DynamoDB)\n\nData Warehouses and Lakehouse platforms\n\nObject Storage solutions\n\nDataOps & Software Engineering\nGit and Git-based workflows\n\nCI/CD pipelines\n\nInfrastructure as Code (Terraform, CloudFormation, Bicep)\n\nContainerization technologies (Docker)\n\nKubernetes fundamentals\n\nAutomated testing principles for data pipelines\n\nNice to have:\nAzure, Databricks or Snowflake certifications.\n\nData Governance & Observability experience with: Data Catalogs, Metadata Management, Data Lineage;\n\nData Quality frameworks\n\nObservability platforms\n\nMaster Data Management concepts Experience working in Data Mesh environments.\n\nExperience supporting AI/ML platforms.\n\nExperience with Data Governance programs.\n\nExperience in FinOps and cloud cost optimization.\n\nExperience mentoring junior engineers and contributing to engineering communities of practice.\n\nWhat’s in it for you:\nHybrid way of working with flexibility and balance\nOpportunity to work on international projects\nPossibility to explore different roles and career paths\nContinuous learning and development opportunities\nA growing AI-driven environment, working with the latest technologies\nA set of benefits and perks to support your day-to-day\nA place for everyone\nAt Celfocus, we are committed to cultivate a diverse and inclusive workplace. As an equal-opportunity employer, we welcome applicants of all backgrounds, gender identities, and abilities.\nIf you require any adjustments during the selection process, please inform our Talent Acquisition Team.","description_format":"text","description_chars":6518,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Continuous learning"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-11-20T15:01:06Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":43,"reasons":["conf:3","win:early"],"computed_at":"2026-11-20T18:26:24Z"},"pay":null,"html_url":"https://alion.io/job/celfocus-data-engineer-azuredatabricks","json_url":"https://alion.io/job/celfocus-data-engineer-azuredatabricks.json","meta":{"generated_at":"2026-11-20T18:26:24Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":23261,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-21T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":54052},"rest":"https://alion.io/mcp/rest/get_company?id=54052"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcelfocus-data-engineer-azuredatabricks"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcelfocus-data-engineer-azuredatabricks"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcelfocus-data-engineer-azuredatabricks"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/celfocus-data-engineer-azuredatabricks\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcelfocus-data-engineer-azuredatabricks"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4696913}},{"tool":"search_jobs","arguments":{"company":"celfocus.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4696913},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}