{"id":4320913,"url":"https://alion.io/job/coffeebeans-consulting-senior-data-engineer-2","title":"Senior Data Engineer","company":{"id":3808032,"name":"Coffeebeans Consulting","domain":"coffeebeans.io","url":"https://alion.io/company/coffeebeans-consulting","size_band":null,"is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":7,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon S3","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GitHub Actions","optional":false},{"name":"IAM","optional":false},{"name":"Jenkins","optional":false},{"name":"MySQL","optional":false},{"name":"PostgreSQL","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"dbt","optional":true},{"name":"Flink","optional":true},{"name":"Snowflake","optional":true}],"status":"live","first_seen_at":"2026-11-14T17:26:26Z","employer_posted_date":null,"last_verified_at":"2026-11-14T17:26:26Z","board_verified":false,"closed_at":null,"days_open":0,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":0},"description":"About the Company\n\nLocation: Bangalore\n\nWorkMode: Hybrid\n\nExperience: 7–10 years\n\nJob Type: Full-time\n\nAbout the Role\n\nWe're looking for seasoned data engineers who can take end-to-end ownership of complex data engineering engagements—right from architecture and design to development and delivery. You'll work with client stakeholders, lead data initiatives, and guide teams in building modern, scalable data platforms that support business intelligence, advanced analytics, and AI/ML applications.\n\nKey Responsibilities\nDesign and implement enterprise-grade Databricks Lakehouse architectures using Delta Lake and Unity Catalog.\nBuild scalable batch and real-time data ingestion pipelines using Lakeflow Connect, SDP, Auto Loader, Spark, and Kafka.\nDesign and implement CDC architectures using Debezium, Kafka/Kafka Connect, and relational databasessuch as PostgreSQL, MySQL, SQL Server, and Oracle.\nImplement streaming and event-driven data pipelines using Kafka, Spark Structured Streaming, and related technologies.\nDesign and manage schema evolution and data contracts using Karapace / Schema Registry.\nImplement centralized governance using Unity Catalog, including catalogs, schemas, RBAC, row/column-level security, lineage, and data access policies.\nDevelop metadata-driven ingestion frameworks, data quality, reconciliation, profiling, and observability solutions.\nDesign Bronze, Silver, and Gold data layers and appropriate data modeling strategies for analytical workloads.\nEstablish engineering best practices covering CI/CD, testing, deployment, monitoring, logging, and operational support.\nUse Databricks Asset Bundles (DAB) and CI/CD tools such as Jenkins/GitHub Actions for automated deployment.\nWork with cloud services such as AWS S3, IAM, networking, monitoring, and security services.\nLead technical discussions with clients, translate business requirements into technical solutions, and drive architecture decisions.\nTroubleshoot complex data engineering, CDC, streaming, performance, and production issues.\nMentor engineers and provide technical direction across data engineering initiatives. Must-Have Skills\nStrong hands-on experience with Databricks and Lakehouse architecture.\nAdvanced Python and SQL skills.\nStrong expertise in Apache Spark / PySpark and distributed data processing.\nHands-on experience with Unity Catalog and Delta Lake.\nExperience with Lakeflow Connect, SDP / Spark Declarative Pipelines, and Auto Loader.\nStrong understanding of CDC architectures using Debezium and Kafka.\nHands-on experience with Kafka / Kafka Connect.\nExperience with Karapace or Schema Registry and schema evolution.\nStrong understanding of ETL/ELT, data modeling, data warehousing, streaming, and data integration patterns.\nExperience with production-grade data pipelines and orchestration.\nStrong understanding of cloud-native data services, particularly AWS.\nExperience with CI/CD and Databricks Asset Bundles (DAB).\nExperience leading technical implementations and working directly with business/client stakeholders.\nGood to Have\nExperience with Snowflake and dbt.\nExperience with Apache Flink or other real-time processing frameworks.\nExperience implementing data governance, lineage, security, data quality, and observability.\nExperience with AWS S3, IAM, Glue, MSK/Kafka, and cloud networking.\nExperience designing metadata-driven data platforms.\nExperience with AI/ML data platforms and GenAI workloads.\nDatabricks certifications, particularly Databricks Certified Data Engineer Professional.\nAWS Data Engineering/Data Analytics certifications. Other Expectations\nStrong ownership and problem-solving mindset.\nAbility to balance hands-on engineering with architecture and technical leadership.\nStrong client-facing and communication skills.\nAbility to mentor and guide engineering teams.\nWillingness to adapt to new technologies and client environments.\nWillingness to travel within India and internationally for short/medium-term client assignments.\n\nKindly drop your resume ([HIDDEN TEXT])","description_format":"text","description_chars":4019,"description_truncated":false,"requirements":{"experience_years_min":7,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","Data Engineering & Migration Services"],"lifecycle":[{"event":"open","at":"2026-11-14T20:10:00Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":16,"reasons":["seen:0","win:early"],"computed_at":"2026-11-15T00:27:12Z"},"pay":null,"html_url":"https://alion.io/job/coffeebeans-consulting-senior-data-engineer-2","json_url":"https://alion.io/job/coffeebeans-consulting-senior-data-engineer-2.json","meta":{"generated_at":"2026-11-15T00:27:12Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":820,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3808032},"rest":"https://alion.io/mcp/rest/get_company?id=3808032"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcoffeebeans-consulting-senior-data-engineer-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcoffeebeans-consulting-senior-data-engineer-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcoffeebeans-consulting-senior-data-engineer-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/coffeebeans-consulting-senior-data-engineer-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcoffeebeans-consulting-senior-data-engineer-2"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4320913}},{"tool":"search_jobs","arguments":{"company":"coffeebeans.io","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4320913},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}