Data Engineer
Andersen is hiring a Data Engineer for a project modernizing a cloud-based quality platform and building scalable data solutions to support analytics, security, and AI capabilities.
The customer is a technology company developing scalable software solutions for organizations across multiple industries. Its products help businesses improve operational efficiency, optimize workflows, and support digital transformation through modern technologies and data-driven approaches. The company focuses on delivering reliable, user-oriented solutions that enable customers to respond effectively to changing business and technology requirements.
The project is focused on modernizing a cloud-based quality intelligence platform by enhancing its API ecosystem, security, data architecture, and AI capabilities. It includes implementing secure public API access, reengineering the data warehouse, improving automated testing and CI processes, and building an agent orchestration platform while supporting SOC 2 compliance.
Responsibilities:
Monitoring and troubleshooting the batch pipelines day to day - the panel named this as the first responsibility: check every run, and when something fails, debug it ("not really glamorous work at first").
Tracing failures through the medallion layers (bronze/silver/gold), distinguishing source-side changes from pipeline bugs, fixing what's understandable, and validating in a test environment before production.
Setting up failure alerting - e.g. sensor notifications into a teams channel showing whether a batch succeeded or failed, with the error surfaced.
Working toward making the refresh cadence of the data more frequent (near-real-time) through incremental-load strategies - named by the panel as a goal the role will grow into.
Handling requests to ingest new data and add new sources independently: setting up connections and credentials, landing data in S3, deciding on the Redshift schema and target tables, and granting IAM access to downstream consumers.
Extracting from the platform's sources (Postgres, MySQL, Google Sheets, Salesforce, HR software) via Glue, load into Redshift, and do transformations in dbt inside the warehouse, feeding the published layer.
Provisioning data to ML engineers and analysts on request (creating views / materialized views, managing access), shaping it around what they need to see.
Reviewing PRs and mentoring junior engineers - checking naming/readability, reuse, and that added functionality doesn't blow the ETL's runtime budget.
Onboarding well enough to review PRs and troubleshooting in a way that matches the platform's overall architecture and modern philosophy - not just patch the error.
Must-haves:
Commercial experience in Data Engineering, owning end-to-end ELT/ETL pipelines and cloud data-warehouse architecture for 4+ years.
Expert-level SQL skills.
Production experience with Amazon Redshift, including schema design, distribution/sort keys, WLM, query tuning, and reducing refresh latency on a live warehouse.
Strong MPP data warehouse experience; Snowflake and/or Azure Synapse SQL Pool experience is valued but does not fully substitute for Redshift experience.
Experience with incremental loads and CDC, including watermarks, SCD2, change-data-capture, and moving from full rebuilds toward near-real-time processing.
Experience re-architecting legacy ETL pipelines and data warehouses, including migration from script-based or on-prem legacy systems to scalable cloud platforms with minimal disruption.
Experience using MySQL as a source system, including incremental extraction from relational OLTP sources; MariaDB familiarity.
Experience with AWS S3 and IAM for data landing and access management.
Production experience with Airflow or equivalent orchestration tools and dbt.
Ability to work fully independently as the sole owner of a platform, including identifying and prioritizing technical debt without formal ticket tracking.
Strong monitoring and failure recovery practices, including checking pipeline runs, debugging failures, and maintaining a healthy batch platform.
Level of English - from Upper-Intermediate and above.
Nice-to-haves:
CDC tooling (e.g. Debezium / binlog-based) in production.
Mentoring junior engineers; PR-review rigor (naming, readability, reuse, runtime budgets).
Reasons why this job would be interesting to you:
Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc..
The opportunity to change the project and/or develop expertise in an interesting business domain.
Job conditions - you can work both fully remotely and from the office or can choose a hybrid variant.
Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
The opportunity to earn up to an additional 1,000 EUR per month, depending on the level of expertise, which will be included in the annual bonus, by participating in the company's activities.
Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
Certification compensation (AWS, PMP, etc).
Referral program.
Private health insurance and sports compensation, depending on the type of employment.
Your personal data is protected in accordance with GDPR regulations. Learn more:
https://andersenlab.com/privacy-policy/pl
Join us!