Senior Data Engineer

Description

Prisma Photonics, a fast-growing startup, transforms infrastructure monitoring with optical fibers. We eliminate the need for extra sensors by offering sensor-free solutions to monitor electrical power grids and oil & gas pipelines across thousands of kilometers.

Our fiber-sensing technology, integrated with AI and machine learning, enables our customers to achieve environmental and renewable energy targets, ensuring smooth utility operations on their path to net-zero emissions.

We are looking for the best minds and spirits to join us in our journey. We know our product is only as great as the individuals building the hardware and software and harnessing data for good causes. Being a great team member means being eager to learn and grow, able to challenge while accepting being challenged, and working for the team and the product with enthusiasm and passion.

We are now hiring a talented, self-driven and passionate Senior Data Engineer to build and maintain optimized and highly available data pipelines that facilitate deeper analysis and reporting.

What You’ll Do

· Design data collection processes including textual and multi-dimensional data.

· Implement data searches and access APIs, between edge computing, cloud stack.

· Develop and optimize the data and metadata storage and databases for our AI and Algorithms to access train and test data efficiently.

· Analyze existing data schemes and processes, suggest and implement ways to improve them.

· Analyze cloud data usage and optimize cost structures while maintaining low latency access using common and proprietary services.

· Work closely with Software, AI and Algorithms teams on system integration, on-going data requests, data access and more.

What You Bring

· 5+ years of professional experience in data engineering or related roles, including hands-on experience in designing and implementing data pipelines, using orchestration frameworks like Apache Airflow, Prefect, etc

· Experience in event driven models and async messaging (kafka, rabbitmq)

· 5+ years of experience and deep understanding of SQL and NoSQL DBs (MySql, Elastic, MongoDB, Postgres, clickhouse).

· 5+ years of SW development, 2+ are Python.

· Full control of version control systems (e.g. Git), containerization (e.g. Docker), and CI/CD pipelines.

· Strong collaborator with teams and peers

· Innovative with a growth mindset.

Great to Have

· Experience with streaming technologies, unstructured data and different data types (other than text and tables).

· Experience with Hadoop, Map Reduce, Spark, or other distributed computing platform.

· Experience with Kubernetes based systems.

· Experience with Data visualization and analysis tools such as - Grafana, ELK, Kibana, etc.