Senior Data Engineer

תאור משרה

:Who We Are
OriginAI, is a large AI research center that was established to address challenges at the forefront of AI research at national scale. The research is focused in the areas of Vision, Speech and NLP with the aim of producing world-class groundbreaking capabilities in these areas
We work in hybrid mode and our new offices located near Tel-Aviv’s Savidor train station

:Job Description
We are seeking a highly skilled and motivated Data Engineer to join our team. The ideal candidate will have expertise in designing and optimizing large-scale data pipelines using RabbitMQ or Kafka, managing vector databases such as Vespa.AI, Elasticsearch, or other leading solutions, and developing Python-based workloads that integrate machine learning models. The systems will handle billions of items and operate in large cluster environments

:Responsibilities
Design, implement, and optimize data pipelines using RabbitMQ or Kafka to handle billions of items
Ensure scalability, fault tolerance, and high-throughput processing capabilities in large clusters
Develop Python-based ML modules integrated with the data pipelines
Set up, maintain, and optimize a vector database (e.g., Vespa.AI, Elasticsearch, or similar) in a distributed cluster environment. Manage database performance, backups, and resource allocation to support billions of data items
Monitor and troubleshoot system performance to ensure operational excellence
Working with Dockerized environments

:Requirements
Proven experience with RabbitMQ or Kafka for building robust, large-scale data pipelines
Proficiency in Python, with experience in creating adapters and integrating ML models
Hands-on experience managing and optimizing vector databases such as Vespa.AI, Elasticsearch, or comparable solutions
Expertise in working with distributed systems and large clusters
Strong problem-solving skills with a focus on system performance and reliability at scale