Data Engineer (Salesforce Focus) 2147

Position: Data Engineer (Salesforce Focus)

Location: Remote (Aligned with US Central Time)

Contract Type: Full-time vendor (Contract directly with In All Media)

Time Zone Alignment: US Central Time (CT)

About In All Media

In All Media is a global technology and design firm focused on building impactful digital solutions through remote, distributed teams globally. We partner with international clients across industries, providing long-term technical expertise, product innovation, and team augmentation.

Contract Details: This is a 100% remote contractor position. Your contract and engagement will be directly managed with In All Media.

Project Overview

In this role, you will drive critical data platform migration, modernization, and integration initiatives across large-scale enterprise environments with a core focus on Salesforce data flows. You will architect, refactor, and operate robust data pipelines-transitioning legacy Hive table structures to modern Apache Iceberg formats, migrating and scaling Apache Airflow DAGs, and establishing cross-regional Kafka event streams. Operating within an AWS-native cloud environment and Snowflake data warehouse, you will ensure high-throughput processing, optimal query performance, and strict data quality standards throughout the migration lifecycle.

Key Responsibilities

• Dataset & Pipeline Migration: Execute the end-to-end migration of mission-critical datasets, legacy data workflows, and orchestrated DAGs to modernized cloud infrastructure.
• Lakehouse Modernization (Hive to Iceberg): Convert legacy Hive table architectures to Apache Iceberg format on AWS S3, improving partition evolution, query performance, and metadata management.
• Distributed Streaming Infrastructure: Reconfigure, validate, and manage Apache Kafka connections, brokers, and topics across multiple AWS regions to support continuous data streaming.
• Snowflake Modeling & Query Optimization: Adapt, model, and tune data pipelines directly in Snowflake, leveraging advanced SQL techniques (CTEs, window functions, complex joins) to optimize compute utilization and latency.
• Workflow Orchestration & Maintenance: Author, schedule, and actively debug Apache Airflow DAGs, resolving upstream failures, optimizing sensor dependencies, and preventing pipeline bottlenecks.
• Data Quality & Testing Strategy: Define and execute structured data reconciliation and validation strategies to guarantee zero data loss, schema consistency, and end-to-end data integrity across migrations.
• Independent Root-Cause Remediation: Autonomously diagnose, profile, and resolve complex performance issues across distributed Spark/PySpark execution jobs, Kafka consumer groups, and AWS storage layers.

Must-Have Skills

• Senior Data Engineering Background: 6+ years of hands-on data engineering experience designing, deploying, and maintaining distributed, production-grade data pipelines on petabyte-scale datasets.
• Core Programming & Advanced Querying: High proficiency in Python (pipeline scripting, data manipulation) and advanced SQL (complex CTEs, window functions, schema design, and query optimization).
• Distributed Processing & Big Data: Proven production expertise with Apache Spark / PySpark for batch and streaming transformations.
• Orchestration & Streaming: Strong hands-on experience authoring and maintaining complex DAGs in Apache Airflow and configuring cross-regional event pipelines with Apache Kafka.
• Modern Lakehouse & Cloud Storage: Direct experience managing AWS S3 data lakes utilizing Apache Iceberg and Parquet formats.
• Cloud Data Warehousing: Hands-on experience with Snowflake schema modeling, warehouse configuration, and query tuning.
• Autonomous Debugging: Demonstrated capability to independently troubleshoot distributed pipeline execution failures, network latencies, and serialization bottlenecks.
• Enterprise Environment Compliance: Must be able to operate within an enterprise AWS WorkSpace (VDI) environment and complete a mandatory AWS network health-check verification (with passing screenshot required during application).

Nice-to-Have Skills

• Enterprise Snowflake Migrations: Hands-on experience migrating legacy on-prem or cloud data warehouses to Snowflake.
• Airflow Platform Upgrades: Practical experience handling platform-level Airflow migrations (e.g., major version upgrades, executor switches to Celery/Kubernetes).
• AI-Assisted Development: Experience utilizing AI-assisted software development tools (e.g., GitHub Copilot, Cursor) to optimize data modeling, pipeline scripting, and debugging.
• Salesforce Domain Experience: Exposure to Salesforce data architectures, object modeling, and CDC/API ingestion mechanisms.

Time Zone & Collaboration

This role requires full-time business-hour alignment with US Central Time (CT) to ensure real-time collaboration with US engineering leadership, active participation in daily standups, and live architectural syncs.

Language

All interviews, technical documentation, architectural reviews, daily team syncs, and operational communications will be conducted in English.