{"id":3480691,"url":"https://alion.io/job/cyble-data-engineer-senior-data-engineer","title":"Data Engineer / Senior Data Engineer","company":{"id":2698,"name":"Cyble","domain":"cyble.com","url":"https://alion.io/company/cyble","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Work at a Startup","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":20000,"max":65000,"currency":"USD","period":"year","gross":null,"usd_annual":65000},"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Airflow","optional":false},{"name":"Amazon ECS","optional":false},{"name":"Amazon EKS","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache Hudi","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Claude Code","optional":false},{"name":"Copilot","optional":false},{"name":"Cursor","optional":false},{"name":"Databricks","optional":false},{"name":"Discord","optional":false},{"name":"ElasticSearch","optional":false},{"name":"Embeddings","optional":false},{"name":"ETL/ELT","optional":false},{"name":"IAM","optional":false},{"name":"Kubernetes","optional":false},{"name":"OpenSearch","optional":false},{"name":"pgvector","optional":false},{"name":"PostgreSQL","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Reranking","optional":false},{"name":"Shuffle","optional":false},{"name":"Spark","optional":false},{"name":"Telegram","optional":false},{"name":"Trino","optional":false},{"name":"CI/CD","optional":true},{"name":"LLM","optional":true},{"name":"Model Context Protocol","optional":true},{"name":"Neo4j","optional":true},{"name":"NER","optional":true},{"name":"OCR","optional":true},{"name":"PaddleOCR","optional":true},{"name":"PaddlePaddle","optional":true},{"name":"Playwright","optional":true},{"name":"Scrapy","optional":true},{"name":"Terraform","optional":true},{"name":"vLLM","optional":true}],"status":"live","first_seen_at":"2026-10-29T20:05:07Z","employer_posted_date":"2026-10-29","last_verified_at":"2026-10-30T22:16:42Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"About the Role\nCyble's threat intelligence platform ingests cybercrime data at petabyte scale: stealer logs, breach dumps, dark web forums and marketplaces, Telegram and Discord channels, ransomware leak sites, and domain intelligence. That data feeds our AI agents and analyst tooling, and it is only useful when it is clean, fresh, deduplicated, and retrievable.\nWe're hiring a Data Engineer /Senior Data Engineer to build and run the Iceberg-based data lake that powers Cyble's AI agents. You'll own pipelines end to end, from raw scraped and leaked data landing in S3 to curated lake tables, OpenSearch indexes, and retrieval-ready datasets that agents and products query. You'll work in a small, low-process team where engineers own their domains, ship directly, and use AI coding tools as a normal part of the workflow.\nWhat You’ll Do at Cyble:\nBuild and operate the data lake powering Cyble's AI agents: ingestion, Data modeling, deduplication, freshness, lineage, and retrieval-ready outputs.\nDesign and run PySpark pipelines on EMR over Apache Iceberg on S3 (Glue Data Catalog), including schema evolution, partitioning, compaction, snapshot expiry, and small-file control.\nIngest scraped, crawled, and leaked data from heterogeneous sources (HTML, JSON, CSV, text dumps, archives, feeds, APIs) and handle the realities: encoding issues, schema drift, malformed records, duplicates, and re-crawled content.\nBuild and tune OpenSearch/Elasticsearch indexing pipelines: mappings, analyzers, shard and index lifecycle strategy, bulk ingestion, and query performance.\nBuild the data side of RAG and agent workloads: chunking, embedding generation, vector and hybrid (keyword + kNN) indexing, and datasets the agents can trust.\nRun streaming and event-driven pipelines on Kafka; orchestrate batch workloads with Airflow.\nDeploy and operate data workloads on Kubernetes (EKS) and ECS, including Spark jobs, Trino, workers, and supporting services.\nOwn lake and pipeline reliability: monitoring, alerting, data quality checks, backfills and reprocessing, retention, and storage/compute cost control.\nUse AI coding assistants and agents (Claude Code, Cursor, Copilot, or similar) daily to build, refactor, test, debug, and document, and review AI-generated code critically before it reaches production.\nWork with security researchers, data scientists, and product engineers on data models for new intelligence products.\nWhat You’ll Need:\nBachelor's degree in computer science, engineering, or equivalent experience.\n3+ years building production data pipelines in Python and Spark.\nDeep Spark knowledge on AWS EMR: shuffle, skew, partitioning, memory tuning, AQE, and cost-aware cluster sizing.\nHands-on experience with Apache Iceberg (or Delta/Hudi with willingness to go deep on Iceberg): table design, partition evolution, compaction, snapshot management, time travel.\nExperience building and operating a data lake or lakehouse that serves downstream AI/ML, search, or agent workloads.\nStrong OpenSearch/Elasticsearch experience: pipeline integration, index design, and performance tuning at scale.\nExperience ingesting and normalizing scraped or semi-structured data at scale, including deduplication and schema-drift handling.\nWorking knowledge of the AWS data stack (S3, Glue, EMR, Lambda, IAM) and containerized deployment on EKS and/or ECS.\nHands-on with PostgreSQL (indexing, query plans, large-table performance) and MongoDB, with the judgment to choose between relational, document, and search stores.\nWorking understanding of RAG architectures: embeddings, vector stores (OpenSearch kNN, pgvector, or similar), chunking strategies, hybrid retrieval, reranking.\nExperience with Kafka (or similar) and Airflow in production.\nHands-on experience using AI coding tools in real data engineering work, with a track record of catching their mistakes.\nAbility to explain technical trade-offs clearly to non-technical stakeholders, and comfort owning a domain end to end in a lean team.\nBonus Points If You Have:\nTrino/Presto on Iceberg, especially on Kubernetes.\nAI/agent-based context extraction in pipelines: LLMs or NER models (e.g., GLiNER) for entity extraction, classification, and enrichment, and their cost and accuracy at scale.\nLLM serving and inference (vLLM or similar) and evaluating retrieval or extraction quality (recall, precision/recall on labeled samples).\nOCR and document extraction (Tesseract, PaddleOCR, Textract, or vision-language models) for images, scans, and PDFs.\nBuilding data access layers for agents (MCP servers, retrieval APIs).\nGraph processing (GraphFrames, Neo4j) for entity relationship mapping.\nWeb scraping frameworks (Scrapy, Playwright) and anti-bot handling, or working downstream of such systems.\nInfrastructure-as-code (Terraform) and CI/CD for data pipelines.\nPrior exposure to cybersecurity or threat intelligence data.\nIf you like working in an inclusive environment, you want to advance your career quickly, and your opinion is valued, look no further than Cyble, Inc. We are young, hungry, and ready to impact the cyber security landscape!\nCyble, Inc. takes into consideration an individual’s skillset, experience and location in making final salary determination.\nAll qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected Veteran status age, or genetics, or any other characteristic protected by law.\nCV Shortlist by the Hiring Panel\n\nCognitive Assessment via a platform called Xobin - It’s a 60-minute assessment, which is a mandatory step in our recruitment process for all roles.\n\nPanel Discussions - Typically comprising a minimum of three interview rounds, along with a role-based assignment.","description_format":"text","description_chars":5787,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Threat Intelligence","Information Security","Cybersecurity AI"],"lifecycle":[{"event":"open","at":"2026-10-29T20:05:07Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":36,"reasons":["conf:1","win:early"],"computed_at":"2026-10-30T05:48:23Z"},"pay":{"stated_usd_annual":65000,"is_top_pay":false},"html_url":"https://alion.io/job/cyble-data-engineer-senior-data-engineer","json_url":"https://alion.io/job/cyble-data-engineer-senior-data-engineer.json","meta":{"generated_at":"2026-10-31T01:51:52Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":2555,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-01T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2698},"rest":"https://alion.io/mcp/rest/get_company?id=2698"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcyble-data-engineer-senior-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcyble-data-engineer-senior-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcyble-data-engineer-senior-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cyble-data-engineer-senior-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcyble-data-engineer-senior-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3480691}},{"tool":"search_jobs","arguments":{"company":"cyble.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3480691},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}