{"id":1369722,"url":"https://alion.io/job/norfolk-southern-lead-data-engineer-databricks-big-data","title":"Lead Data Engineer (Databricks, Big Data)","company":{"id":1797468,"name":"Norfolk Southern","domain":"norfolksouthern.com","url":"https://alion.io/company/norfolksouthern","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"SuccessFactors","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Atlanta, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":115000,"max_usd":229000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":315},"experience_years_min":10,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Amazon Kinesis","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Azure","optional":false},{"name":"BigQuery","optional":false},{"name":"Cassandra","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Dimensional Modeling","optional":false},{"name":"Erwin","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Google BigQuery","optional":false},{"name":"HBase","optional":false},{"name":"IAM","optional":false},{"name":"IBM MQ","optional":false},{"name":"Kanban","optional":false},{"name":"Machine Learning","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Rest API","optional":false},{"name":"Scala","optional":false},{"name":"Scrum","optional":false},{"name":"Snowflake","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"COBOL","optional":true}],"status":"live","first_seen_at":"2026-09-17T00:00:00Z","employer_posted_date":"2026-09-17","last_verified_at":"2026-10-05T21:40:12Z","board_verified":true,"closed_at":null,"days_open":19,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":19},"description":"Requisition 40310: B5 Lead Data Engineer (Databricks, Big Data)\nA resume helps you stand out to hiring managers and recruiters; your resume communicates your experience and your brand. While it is not required, we encourage you to include an up-to-date resume along with a completed job application to give you the best opportunity to be considered. A complete resume helps us to better understand your unique background, relevant experiences, and passions. We look forward to learning about you.\nNorfolk Southern offers a unique opportunity to be part of our proud legacy that spans nearly 200 years. We are a customer-centric, operations-driven team dedicated to advancing safety, serving communities, and driving innovation for tomorrow's rail. As part of Norfolk Southern, you’ll join a collaborative team where there are opportunities for growth across the organization. We are building a culture where everyone can thrive by owning and driving exceptional results, being humble and leading with trust, serving our customers with excellence, and collaborating and coaching to win.\nJob Description\nNorfolk Southern Corporation is seeking a Lead Data Engineer who enjoys collaborating across the organization to deliver reliable, scalable data solutions. Join a skilled team focused on modern data platforms, where you will lead the definition of architecture, help shape and operate our Databricks-based lakehouse and streaming analytics capabilities, and guide the team through technical delivery.\nThis role blends strong business and analytical judgment with hands-on engineering and technical leadership: you will partner with stakeholders and the engineering team to understand problems, translate them into clear requirements, define and drive pragmatic architectures, and deliver production-grade ingestion and transformation pipelines - with ownership through deployment and steady-state operations. You will also work closely with the team to proactively identify and remove technical blockers that stand in the way of delivery.\nIn this role, you will work with teams across the company to clarify Data and Analytics needs, perform structured requirements discovery alongside Data Modelers, align with BI development on scope and estimates, and lead and guide delivery through completion while promoting engineering quality, consistency, and best practices across the team.\nJob Responsibilities\nLead the definition of data architecture in partnership with the team and stakeholders, setting technical direction for the lakehouse, ingestion, and transformation patterns.\nIdentify, escalate, and remove technical blockers for the team - unblocking delivery through hands-on problem solving, design decisions, or coordination with other teams.\nLead the design and development of enterprise data models (conceptual, logical, and physical) across domains, ensuring they align with lakehouse and pipeline architecture.\nDefine and enforce data modeling standards, best practices, and governance processes, leveraging tools such as Erwin.\nDefine data requirements, gather, and wrangle large scale of structured and unstructured data, and validate data by running various data tools in the Data Environment.\nSupport the standardization, customization and ad-hoc data analysis, and will develop the mechanisms to ingest, analyze, validate, normalize and clean data.\nCreate data policy and develop interfaces and retention models which require synthesizing or anonymizing data.\nImplement statistical data quality procedures on new data sources, and by applying rigorous iterative data analytics, supports Data Scientists and analytics and insights creation in data sourcing and preparation to visualize data and synthesize insights of commercial value.\nDevelops and maintains data engineering best practices and contributes to Insights on data analytics and visualization concepts, methods and techniques.\nLead Data Engineering team to use state-of-the-art Big Data tools and technologies to build scalable data architecture and performant data pipelines to acquire high-velocity real-time and batch data of different sizes and scales\nWork closely with the data science and business intelligence teams to develop data models and pipelines for research, reporting, and machine learning.\nBuild data pipelines that clean, transform, and aggregate data from disparate sources.\nEmploy a variety of languages and tools (e.g. scripting languages) to marry systems together.\nApply knowledge of Data Architecture components, leads project teams from requirements to implementation.\nEducation Required\nBachelor’s Degree, preferably in Information Systems, Computer Science, Computer Information Systems or related technology field.\nSkills Required\nDatabricks Lakehouse Expertise (Required): 10+ years in data engineering with deep, hands-on experience across the Databricks ecosystem including Spark/PySpark pipelines, Delta Lake (merges, schema evolution), Delta Live Tables, Unity Catalog governance, Genie for self-service analytics, and LakeFlow Designer for visual ETL orchestration.\nStreaming & Cloud Infrastructure: 4+ years building reliable batch/streaming Spark workloads, 3+ years with Kafka (or Confluent) for high-volume event processing, and 3+ years working with AWS analytics services (S3, IAM, Glue/Lambda/MSK).\nData Modeling & SQL: Proven ability to design conceptual, logical, and physical data models with strong governance practices, paired with advanced SQL skills for building reliable, business-ready datasets.\nTechnical Leadership: Track record of leading technical design, mentoring engineers, driving architectural consensus, and unblocking teams on complex data engineering challenges.\nDelivery & Collaboration: Experience delivering ETL/ELT pipelines end-to-end in a lakehouse environment, working within Agile frameworks (Scrum/Kanban/SAFe) to manage iterative delivery and cross-team dependencies.\nSkills Desired\nPrior experience as a technical lead, staff engineer, or similar role with formal responsibility for architecture decisions and team unblocking.\nPython (and/or Scala) and PySpark/Scala-Spark.\nDatabase solutions like Snowflake, Delta Lake or BigQuery.\nNoSQL databases, including HBASE and/or Cassandra.\nAzure, AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue.\nMessaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.\nStrong understanding of Relational & Dimensional modeling.\nExperience with GIT code versioning software.\nExperience with REST API and Web Services.\nWork Conditions\nLocation: Atlanta, GA\nEnvironment: Hybrid (two days in office; three days remote)\nShift Work: No\nOn-Call: Yes\nWeekend Work: No\nCompany Overview\nSince 1827, Norfolk Southern Corporation (NYSE: NSC) and its predecessor companies have safely moved the goods and materials that drive the U.S. economy. Today, it operates a customer-centric and operations-driven freight transportation network. Committed to furthering sustainability, Norfolk Southern helps its customers avoid 15 million tons of yearly carbon emissions by shipping via rail. Its dedicated team members deliver more than 7 million carloads annually, from agriculture to consumer goods, and is the largest rail shipper of auto products and metals in North America. Norfolk Southern also has the most extensive intermodal network in the eastern U.S., serving a majority of the country’s population and manufacturing base, with connections to every major container port on the Atlantic coast as well as the Gulf of Mexico and Great Lakes. Learn more by visiting www.NorfolkSouthern.com.\nAt Norfolk Southern, we believe in celebrating our individuality. By leveraging the unique backgrounds and viewpoints of our employees, we can create a culture of innovation, respect, and inclusion. We know that employees thrive in a workplace where differing viewpoints, ideas, and experiences are freely shared and valued. As such, we encourage all employees to contribute their distinctive skills and capabilities to our organization.\nEqual employment opportunities are available to all applicants regardless of race, color, religion, age, sex, national origin, disability status, genetic information, veteran status, sexual orientation, and gender identity. Together, we power progress.","description_format":"text","description_chars":8287,"description_truncated":false,"requirements":{"experience_years_min":10,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":true},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Transportation & Logistics","Rail Transport","Intermodal Transport"],"lifecycle":[{"event":"open","at":"2026-09-28T03:38:17Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 21 · green card filings: 4","filings_12m":21,"filings_prev_12m":12,"green_card_filings_12m":4,"median_offered_wage_usd":135100,"route":null,"cap_exempt":false,"checked_at":"2026-10-03T21:08:04+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":2}],"liveness":{"score":59,"band":"ok","label":"Likely open","p_open":1,"p_active":0.781,"p_room":0.75,"age_days":18,"expected_fill_days":22,"reasons":["conf:1","velocity","win:late"],"computed_at":"2026-10-05T05:45:15Z"},"pay":null,"html_url":"https://alion.io/job/norfolk-southern-lead-data-engineer-databricks-big-data","json_url":"https://alion.io/job/norfolk-southern-lead-data-engineer-databricks-big-data.json","meta":{"generated_at":"2026-10-06T00:28:47Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler","counted_by":"address","units_charged":1,"used_today":577,"day_limit":5000,"remaining_today":4423,"minute_limit":60,"resets_at":"2026-10-07T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1797468},"rest":"https://alion.io/mcp/rest/get_company?id=1797468"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fnorfolk-southern-lead-data-engineer-databricks-big-data"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fnorfolk-southern-lead-data-engineer-databricks-big-data"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fnorfolk-southern-lead-data-engineer-databricks-big-data"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/norfolk-southern-lead-data-engineer-databricks-big-data\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fnorfolk-southern-lead-data-engineer-databricks-big-data"}]}