{"id":2563710,"url":"https://alion.io/job/wiley-principal-engineering-lead-content-data-pipeline","title":"Principal Engineering Lead — Content & Data Pipeline","company":{"id":1990,"name":"Wiley","domain":"wiley.com","url":"https://alion.io/company/wiley","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":null},"role":"Management","role_family":"Management","seniority":"lead","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hoboken, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":140000,"max":200733,"currency":"USD","period":"year","gross":null,"usd_annual":200733},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon ECS","optional":false},{"name":"Amazon EKS","optional":false},{"name":"Amazon EventBridge","optional":false},{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"CI/CD","optional":false},{"name":"Dagster","optional":false},{"name":"DynamoDB","optional":false},{"name":"IAM","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"SQL","optional":false},{"name":"AI Agents","optional":true},{"name":"Amazon Kinesis","optional":true},{"name":"Apache Kafka","optional":true},{"name":"GCP","optional":true},{"name":"Google BigQuery","optional":true},{"name":"Kubernetes","optional":true}],"status":"live","first_seen_at":"2026-10-14T21:26:20Z","employer_posted_date":"2026-10-14","last_verified_at":"2026-10-22T17:44:10Z","board_verified":true,"closed_at":null,"days_open":8,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":8},"description":"We believe in bold ideas, diverse perspectives, and the drive to transform knowledge into impact. Here, your curiosity fuels progress, your voice shapes innovation, and your ambition helps redefine what’s possible within science and learning. We are a culture that obsesses over impact, challenges, and drives what’s next to power infinite possibilities for our customers, colleagues and society at large.\nAbout the Role:\nAbout the role\nThis role leads Wiley's Content & Data Pipeline function - the platform that takes scholarly content (journals, articles, books) from a wide mix of sources through ingestion, processing, and enrichment, and delivers it to internal AI systems, RAG pipelines, and external customers under the right entitlements. It is a hybrid role: you set technical direction and stay hands-on with architecture and code where it matters, and you also manage and grow a team of engineers who own the day-to-day pipeline work.\nThe role sits alongside an existing dev lead who runs the current content ingestion and delivery pipeline; you take end-to-end ownership above that, expanding the platform to cover new source types, new delivery surfaces (especially internal AI / RAG and licensed external delivery), and higher standards of quality, freshness, and observability.\nThe mandate spans the full content lifecycle - ingestion, transformation, enrichment, metadata management, rights-aware delivery, and vendor / supplier integration - across scholarly publications and books. Stakeholders include product, business, licensing partners, and vendors, and part of the job is turning their needs into a coherent, prioritized platform roadmap.\nWe are looking for someone who leads by doing and by coaching - who codes, reviews, unblocks, and mentors, while also owning delivery, and stakeholder alignment for the team.\nWhat you'll do\nTechnical direction and architecture\nOwn end-to-end architecture for the content and data pipeline: ingestion from vendors and public sources, transformation and enrichment, metadata management, and delivery to internal AI / RAG systems and external customers.\nRun and operate the pipeline in production and manage the content lifecycle - corrections, retractions, version updates, metadata deposits and registry syndication, and rights and entitlements enforced at every delivery channels.\nSet standards for pipeline reliability, data quality, freshness monitoring, testing, observability, and CI/CD across the team's services.\nCodify architectural decisions in ADRs; drive convergence on shared patterns across ingestion, processing, and delivery.\nStay hands-on - write code, run architecture reviews, prototype, and unblock the team where needed.\nTeam leadership and delivery\nManage and coach a team of engineers spanning content ingestion, processing, and delivery. \nPartner closely with the existing dev lead on the current pipeline; align on ownership boundaries and coordinate roadmap execution.\nOwn delivery commitments - scope, sequencing, dependencies, and honest tradeoff conversations with stakeholders when reality does not match the plan.\nRun engineering practices that scale: code review discipline, on-call rotation, incident response and postmortems, quality gates in CI.\nStakeholder partnership\nPartner with product and business teams on capability priorities - what content the platform ingests next, what delivery surfaces get built, what quality standards matter.\nOwn vendor and licensing-partner relationships for content sources and delivery pipes - technical integration, data contracts, escalation, renewal input.\nTranslate stakeholder needs (editorial ops, product, licensing) into a coherent platform roadmap; push back honestly when requests do not fit the platform's direction.\nCommunicate technical strategy and tradeoffs upward and outward - not just to peer engineering leaders but to product, business, and executive audiences.\nWhat you'll own\nThe Content & Data Pipeline platform end-to-end - ingestion through delivery - including architecture, standards, delivery SLAs, and data quality guarantees to internal and external consumers.\nThe Content & Data Pipeline team - build and delivery\nVendor and licensing-partner technical relationships for content sources and delivery pipes in scope.\nThe technical roadmap for expanding the platform to new sources, new delivery surfaces, and higher engineering bars, codified in ADRs and communicated to stakeholders.\nMust-have skills\nLeadership\nPlayer-coach engineering leadership at scale - you have led an engineering team while remaining genuinely hands-on with architecture and code. \nCross-team influence with product, business, vendor, and executive stakeholders. Able to communicate technical strategy in language each audience actually uses.\nVendor and partner relationship management - technical integration, contract execution, renewal input, escalation.\nHonest tradeoff communication - you make scope, quality, and timeline conversations concrete rather than optimistic.\nTechnical depth\nDeep production experience designing and operating content or data pipelines end-to-end - ingestion, processing, delivery - at scale.\nAWS at production depth - S3, Lambda, ECS or EKS, SQS / SNS, EventBridge, Step Functions, RDS, DynamoDB. Real IAM and cost fluency.\nPython and SQL at production depth.\nWorkflow orchestration in production - Airflow, Step Functions, Dagster, or equivalent - with real experience in idempotency, backfills, retries, dependency management.\nContent transformation experience - XML processing, schema-driven transformation, structured content formats.\nData quality practice: schema validation, defensive parsing, incremental correctness checks, freshness monitoring, alerting on drift.\nDistributed systems literacy: idempotency, retries, duplicate handling, partial failure, replay.\nReal production experience with API-based and bulk ingestion from external sources - auth, pagination, rate limiting, silent schema changes.\nTesting and CI/CD discipline for pipelines - unit, integration, and data-quality tests in CI; safe deployment patterns.\nObservability practice - logs, metrics, lineage, and the discipline of knowing whether the pipeline is actually healthy vs. just running.\nActive use of AI coding agents in your day-to-day workflow - always with review.\nHugely preferred: content pipeline experience\nScholarly publishing - journals, articles, books - is our domain and our first preference. But the core skill is building pipelines for content, not just data, and that experience carries over from any content-heavy environment.\nYou have built or run pipelines that ingest, process, and deliver content in structured formats - XML, HTML, EPUB, PDF - where structure and metadata matter.\nYou have dealt with content that gets corrected, updated, or pulled after it ships, and built pipelines that handle that cleanly.\nYou have delivered the same content to different consumers in different shapes - feeds, APIs, search, AI systems.\nYou have worked with content that has usage rights or access restrictions attached.\nYou have partnered with editorial, production, or content-operations teams and earned their trust.\nScholarly-specific knowledge - JATS, DOIs, Crossref, retraction handling - is a strong plus. We can teach the acronyms to someone who already has the pipeline instincts.\nNice to have\nScholarly publishing specifics - JATS / BITS XML, DOI infrastructure, Crossref deposits, ORCID / ROR identity, MeSH and similar controlled vocabularies.\nBigQuery / GCP experience - useful for the secondary consumer surface, especially as our AI teams standardise there.\nSpecialised publishing tooling - XSLT at scale, existing content-management systems, or custom XML processing frameworks.\nStreaming ingestion - Kafka, Kinesis, Pub/Sub - for near-real-time content updates.\nVector search and embedding pipelines - useful for the internal AI / RAG delivery surface.\nPrior partnership with ML, AI, or agent teams as a downstream consumer of your platform.\nData catalog and lineage tooling: DataHub, Amundsen, or equivalent.\nExisting relationships in the scholarly publishing ecosystem - vendors, standards bodies, or peer engineering leaders at similar organisations.\nHow we work\nPlayer-coach leadership. We lead by doing and by coaching. Purely delegating leaders and purely heads-down principal engineers are both mismatches for this role.\nContent as a first-class engineering concern. Scholarly content has production standards, versioning semantics, and rights implications that generic data pipelines do not - and we engineer to them.\nEnd-to-end ownership. We own ingestion through delivery. We do not throw work over the wall between ingestion, processing, and delivery teams; we design across them.\nStakeholder partnership. Product, business, vendors, and executives are partners, not audiences. We share evidence and tradeoffs openly, and we push back honestly when needed.\nLearning agility. The scholarly content and AI landscapes both move quickly. We hire leaders who show up curious about new territory rather than defensive of what they already know.\nQuality is a first-class engineering concern. Data-quality bugs are engineering bugs, not “the upstream source's problem.”\nAgentic engineering is a strong expectation, including at leadership levels. We use AI coding agents actively - always with review - and we set the standards our teams follow.\nProduction ownership. Engineers, including engineering leaders, own the platforms they build - including on-call, incident response, and using observability tools to diagnose their own systems.\nWe power infinite possibilities.\nFor more than 200 years, we've transformed knowledge into discoveries that shape the world. Today, our global team of innovators, creators, and experts is driving what's next in science, education, and publishing-creating impact that reaches everywhere.\nWe're not just observers of progress. We're the ones accelerating scientific breakthroughs, advancing learning, and sparking innovation that redefines entire fields and improves lives.\nHere, your talent matters. Your ideas have room to grow. And your work creates breakthroughs that can change everything. \nWiley is an equal opportunity/affirmative action employer. We evaluate all qualified applicants and treat all qualified applicants and employees without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, disability, protected veteran status, genetic information, or based on any individual's status in any group or class protected by applicable federal, state or local laws. Wiley is also committed to providing reasonable accommodation to applicants and employees with disabilities. Applicants who require accommodation to participate in the job application process may contact  for assistance.\nWe are proud that our workplace promotes continual learning and internal mobility. We offer meeting-free Friday afternoons allowing more time for heads down work and professional development, and through a robust body of employee programing we facilitate a wide range of opportunities to foster community, learn, and grow.\nWe are committed to fair, transparent pay, and we strive to provide competitive compensation in addition to a comprehensive benefits package. The range below represents Wiley's good faith and reasonable estimate of the base pay for this role at the time of posting roles in the United Kingdom, Canada, USA, Austria, Czechia, Denmark, France, Greece, Italy, Netherlands, Romania, or Spain. It is anticipated that most qualified candidates will fall within the range, however the ultimate salary offered for this role may be higher or lower and will be set based on a variety of non-discriminatory factors, including but not limited to, geographic location, skills, and competencies.\nWhen applying, please attach your resume/CV to be considered.\nSalary Range:\n140,000.00 USD to 200,733.33 USD...","description_format":"text","description_chars":12007,"description_truncated":true,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Professional development"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["EdTech Platforms","Publishing"],"lifecycle":[{"event":"open","at":"2026-10-14T21:26:20Z"}],"visa":[],"liveness":{"score":85,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.897,"p_room":0.945,"age_days":8,"expected_fill_days":22,"reasons":["conf:12","velocity","win:mid"],"computed_at":"2026-10-23T05:54:15Z"},"pay":{"stated_usd_annual":200733,"is_top_pay":true},"html_url":"https://alion.io/job/wiley-principal-engineering-lead-content-data-pipeline","json_url":"https://alion.io/job/wiley-principal-engineering-lead-content-data-pipeline.json","meta":{"generated_at":"2026-10-23T19:41:05Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":771,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-24T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1990},"rest":"https://alion.io/mcp/rest/get_company?id=1990"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fwiley-principal-engineering-lead-content-data-pipeline"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fwiley-principal-engineering-lead-content-data-pipeline"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fwiley-principal-engineering-lead-content-data-pipeline"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/wiley-principal-engineering-lead-content-data-pipeline\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fwiley-principal-engineering-lead-content-data-pipeline"}]}