Data Engineer (AI Platform/Data Infrastructure)

Responsible for developing and operating a data platform that supports data utilization

Job Description

Established in March 2022 by Marubeni Group, Kodansha Ltd., Shogakukan Inc., and Shueisha Inc., with the aim of transforming Japan's publishing distribution into a sustainable one through the use of DX, this company will provide data collection infrastructure to the system development members of the AI Supply Chain Solutions Division. The division aims to expand to a size of over 50 people in the future.

Job Details

PubteX's data engineers develop and provide data infrastructure for utilizing data for various purposes, such as web applications, planning systems, and machine learning, under the mission of creating valuable data that forms the foundation of publishing distribution and making publishing distribution sustainable by purifying and organizing data across the entire publishing distribution industry.

In this position, as a data platform engineer, you will be responsible for the design and implementation accompanying the renewal of existing systems, and the expansion of the platform in response to new customers and new initiatives.

This position offers experience and growth opportunities from the infrastructure and application layers to the business layer.

About the AI Supply Chain Solutions Division

To realize sustainable publishing distribution, we provide supply chain optimization services utilizing DX (Digital Transformation). While this involves many stakeholders, including publishers, distributors, and bookstores, and requires significant coordination, it's an extremely rewarding position where you can reform the entire supply chain from upstream to downstream. If you're tired of theoretical analysis and want to challenge yourself with a new role, let's work together to transform the industry!

Growth Opportunities

• Many newly created positions available as we are a newly established company!
• Experience the thrill of leading clients to success alongside talented members.
• You can execute end-to-end solutions, from system-based proposals to value creation, to address client challenges.

⇒This opens up career paths to senior management and business leadership roles!

Development Environment

PubteX's Technology Stack:

• Communication: GitHub, Slack
• GCP: BigQuery, AlloyDB, Composer, Cloud Run, Cloud Scheduler, Google Compute Engine, Google Cloud Storage, VPC Service Controls, etc.
• Development Languages: Python, Typescript
• Frameworks: Airflow (Composer), dbt, Dagster, LightGBM, React Router, Hono, FastAPI
• Infrastructure: Terraform, Ansible
• CI/CD: GitHub Actions, Cloud Build

• Scope of Changes to Responsibilities: We will primarily discuss your skills and experience during the interview before offering you a position. Depending on your suitability, we may discuss and potentially change your overall work responsibilities.
• Scope of Changes to Work Location: Either KDX Toranomon 1-chome Building or remote work.

Essential Skills

Must-Have Abilities

• Experience developing and operating data-related products using programming languages (such as Python) on the cloud
• Business-level Japanese language proficiency (JLPT N2 or higher if Japanese is not your native language)

In addition, experience with at least three of the following is required:

• Experience in designing and developing data platforms
• Experience in designing and developing systems using SQL with DBMS such as BigQuery, Redshift, Athena, Synapse, PostgreSQL, MySQL, MariaDB, Oracle, and SQL Server
• Experience in environment setup and development on public clouds (GCP, AWS, Azure)
• Experience in system development using workflow engines such as Apache Airflow, Apache NiFi, Cloud Workflows, and Step Functions
• Experience in database design, construction, and operation (including performance tuning)

Skills that would be a plus if experienced:

• A thorough understanding of relational models and the ability to model data based on them
• Project leader/sub-leader experience
• Experience communicating with stakeholders and defining requirements
• Knowledge of infrastructure
• Extensive knowledge and experience in information science