{"id":4358339,"url":"https://alion.io/job/idbc-senior-data-engineer-2","title":"Senior data engineer","company":{"id":2911996,"name":"IDBC","domain":"idbc.hu","url":"https://alion.io/company/idbc","size_band":null,"is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":"Zoho Recruit","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Budapest, Hungary"],"countries":["HU"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Active Directory","optional":false},{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Azure DevOps","optional":false},{"name":"Azure SQL Database","optional":false},{"name":"CI/CD","optional":false},{"name":"Confluence","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Dimensional Modeling","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GDPR","optional":false},{"name":"Git","optional":false},{"name":"Machine Learning","optional":false},{"name":"MLFlow","optional":false},{"name":"Power BI","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Vault","optional":false}],"status":"live","first_seen_at":"2026-06-19T00:00:00Z","employer_posted_date":"2026-06-19","last_verified_at":"2026-11-16T14:19:06Z","board_verified":true,"closed_at":null,"days_open":150,"trust":{"level":"stale","repost_count":0,"flags":["stale"],"days_open":150},"description":"• Data Pipeline Design & Development (ETL/ELT) o Develop ETL/ELT Pipelines: Build scalable ETL/ELT pipelines using Azure Databricks, integrating multiple data sources like Azure Blob Storage, Azure Data Lake, SQL Databases, etc. o Automation of Data Workflows: Automate data ingestion, transformation, and loading processes through Azure Data Factory (ADF), Databricks workflows, and Azure Functions. o Real-time Data Processing: Implement real-time streaming data pipelines using Databricks Structured Streaming for use cases such as IoT or event-driven architectures.Data Transformation & Modeling o Data Cleaning & Transformation: Leverage Apache Spark in Databricks to process large datasets efficiently, performing data cleansing, transformation, and enrichment. o Data Modeling: Design and implement dimensional data models (e.g., star schema) optimized for performance and querying in Azure Synapse Analytics or other reporting layers. o Delta Lake Implementation: Use Delta Lake for reliable and scalable ACID-compliant data storage and to optimize data for batch and stream processing.\nOptimization & Performance Tuning o Optimize Data Processing: Tune Databricks notebooks and jobs for performance, leveraging Databricks' autoscaling features and optimizing Apache Spark configurations for specific workloads. o Data Partitioning & Indexing: Implement best practices for partitioning large datasets, managing table storage formats (Parquet, Delta), and indexing data for faster querying. o Cluster Management: Manage Databricks clusters (autoscaling, sizing, and costs), ensuring efficient resource utilization on Azure.\nCollaboration & Integration with Azure Services o Azure Integration: Integrate Databricks with other Azure services like Azure Data Lake Storage, Azure SQL, Azure Synapse, Azure Key Vault (for security), and Power BI for seamless data flow and analysis. o Monitoring & Alerts: Set up monitoring, alerting, and logging of data pipelines using tools like Azure Monitor, Databricks Jobs, and Azure Log Analytics to ensure smooth operations.\nData Governance & Security o Data Security: Ensure security measures such as data encryption, role-based access control (RBAC), and compliance with GDPR and other regulations using Azure Active Directory (AAD) and Databricks Secrets. o Data Quality Management: Implement data quality checks in Databricks pipelines, ensuring consistency, accuracy, and validity of data. o Version Control & CI/CD: Use version control systems like Git and implement CI/CD pipelines using Azure DevOps for Databricks notebooks and data workflows.\nCollaboration & Documentation o Cross-functional Collaboration: Collaborate with data scientists, analysts, and business users to develop insights and solutions that drive business objectives. o Documentation: Maintain thorough documentation of data architectures in DF Confluence, processes, and pipelines in Databricks for future scalability and team collaboration.\nAdvanced Analytics & Machine Learning o Machine Learning Integration: Collaborate with data science teams to build and deploy machine learning models using Databricks MLflow and integrate with Azure's machine learning services for operationalization. o Data Exploration: Support exploratory data analysis and business intelligence needs using Databricks Notebooks and integrate with Azure Power BI or other visualization tools.\nRequirements Hands-on experience with Databricks (mandatory) and proven expertise in designing, developing, and maintaining data solutions on the Databricks platform. 6-8+ years of professional experience in Data Engineering Strong SQL expertise Proficiency in Python for data processing, automation, and backend data development. Experience with Microsoft Azure, including cloud-based data services and data platform implementations. Ability to work in modern cloud-based data environments and contribute to the migration and adoption of Databricks as a strategic data platform. Strong analytical and problem-solving skills with a focus on scalable and high-quality data solutions.","description_format":"text","description_chars":4075,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-16T08:03:49Z"}],"visa":[],"liveness":{"score":6,"band":"cold","label":"Long shot","p_open":1,"p_active":0.2,"p_room":0.28,"age_days":150,"expected_fill_days":30,"reasons":["conf:0","agency","win:tail","crowd:"],"computed_at":"2026-11-16T14:38:02Z"},"pay":null,"html_url":"https://alion.io/job/idbc-senior-data-engineer-2","json_url":"https://alion.io/job/idbc-senior-data-engineer-2.json","meta":{"generated_at":"2026-11-16T14:38:02Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":22865,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-17T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2911996},"rest":"https://alion.io/mcp/rest/get_company?id=2911996"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fidbc-senior-data-engineer-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fidbc-senior-data-engineer-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fidbc-senior-data-engineer-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/idbc-senior-data-engineer-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fidbc-senior-data-engineer-2"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4358339}},{"tool":"search_jobs","arguments":{"company":"idbc.hu","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4358339},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}