AI Benchmark Engineer | Native Language Specialist

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a AI Benchmark Engineer | Native Language Specialist based in France.

This freelance opportunity is designed for an experienced software engineer with native-level French expertise who wants to help evaluate the multilingual capabilities of advanced AI systems. You will design rigorous Terminal-Bench tasks that test how coding agents handle French-language prompts, data, encoding, localization, and complex terminal workflows. Your work will combine software engineering, language expertise, benchmark design, and quality assurance. You will create realistic environments and deterministic evaluation systems that distinguish genuine multilingual understanding from solutions that simply rely on English translation. The role involves calibrating task difficulty across different AI model capabilities and analyzing execution results. You will contribute to a rigorous human and AI-assisted quality process designed to make multilingual AI evaluations reliable, fair, and technically meaningful.

Accountabilities

• Design and engineer high-signal Terminal-Bench tasks that evaluate the ability of coding agents and large language models to solve realistic software challenges in French-language environments.
• Build authentic task environments using datasets, files, prompts, and other assets written in French, ensuring that the target language remains present throughout the evaluation rather than being replaced by English translations.
• Identify meaningful failure points in AI systems by developing French-language prompts and scenarios that test multilingual reasoning, software behavior, and language-specific edge cases.
• Develop and validate reference implementations that provide robust solutions against which model-generated outputs can be evaluated.
• Write reliable, deterministic verifier scripts and use rubric-based evaluation only when deterministic verification is not practical or sufficient.
• Analyze execution logs and calibrate task difficulty across levels ranging from Easy to Very Hard using standardized Terminal-Bench configurations and different model tiers.
• Participate in a multi-layer quality assurance process covering task creation, human review, calibration review, and final audit, complemented by automated LLM-based quality checks.
• Review tasks for grammatical accuracy, linguistic authenticity, technical correctness, fairness, reproducibility, and overall benchmark integrity.
• Identify issues in task design, documentation, evaluation logic, or language handling and refine benchmarks to improve their reliability and signal quality.
• Contribute technical and linguistic expertise to help measure AI robustness across multilingual text processing, terminal workflows, and locale-specific software behavior.

Requirements

• At least 5 years of professional experience in software engineering, with a strong track record of delivering technically demanding software projects.
• Proven experience working at leading technology companies and/or graduation from a highly regarded engineering university or equivalent technical background.
• Native or near-native fluency in French, with deep knowledge of French grammar, register, phrasing, and language-specific conventions, combined with strong English proficiency.
• Strong proficiency in Python, standard shell scripting, and data processing workflows.
• Extensive experience with terminal- and CLI-based development environments and a working familiarity with modern coding agents or AI-assisted software development workflows.
• Deep technical understanding of multilingual text-processing challenges, including encoding and decoding, Unicode normalization, locale-dependent behavior, collation, casing, and non-Gregorian or localized date conventions.
• Strong knowledge of text input/output, interoperability between command-line tools and software systems, and safe string handling.
• Understanding of language-specific rendering and typography considerations, including bidirectional or RTL behavior and font fallback where relevant to multilingual artifacts.
• Strong analytical and debugging abilities, with a rigorous approach to reproducibility, verification, and technical quality.
• Ability to work independently as a freelance contributor while consistently meeting agreed quality standards and delivery deadlines.
• Strong attention to linguistic detail, technical accuracy, and consistency when reviewing both AI-generated outputs and benchmark implementations.
• Availability to commit to a minimum of approximately 2 hours per day or 10 hours per week once tasks are accepted, recognizing that overall project availability may fluctuate based on demand.
• Ability to work as an independent contractor and take responsibility for applicable tax obligations.
• Must be legally able to provide freelance services from France and comply with applicable geographic restrictions; contractors in regions subject to international embargoes or sanctions cannot be engaged.

Benefits

• Fully remote freelance work that can be performed from France.
• Flexible scheduling with no fixed working hours or traditional check-in requirements, allowing you to organize your work around your availability.
• Competitive freelance rates with prompt payments and no need to chase invoices.
• Opportunity to contribute directly to cutting-edge AI evaluation and multilingual language technology.
• Work on technically challenging projects involving large language models, coding agents, terminal environments, and multilingual software engineering.
• Access to a global community of linguists, software engineers, subject-matter experts, and language professionals.
• Exposure to diverse AI projects that can strengthen your technical portfolio and broaden your experience across domains.
• Opportunity to apply both your French language expertise and software engineering skills to problems with real-world impact.
• Project availability may fluctuate, making this opportunity better suited to supplemental income rather than a guaranteed full-time or primary income source.
• As an independent contractor, benefits such as health insurance, paid time off, and retirement contributions are not provided.
• Work availability and hours are not guaranteed, although accepted assignments require reliable availability, quality delivery, and adherence to agreed deadlines.