C
Hiring companyClearwater
Active listing
Senior Security Data Engineer
Pune, India Full-time On-site
About the role
Senior Security Data Engineer responsible for building and maintaining high-throughput streaming pipelines that ingest logs from firewalls, cloud services, and endpoints. Designs parsers to normalize raw logs into standard schemas (OCSF/ECS), optimizes cost by routing high-value data to SIEM and bulk data to low-cost object storage, and prepares data for AI/ML detection models and advanced analytics. Requires strong Python, SQL, Kafka, regex, and data lake expertise.
What you’ll do
- Design and maintain streaming pipelines for log ingestion.
- Develop parsers to convert raw logs into standardized schemas.
- Implement routing logic to balance cost and performance between SIEM and storage.
- Prepare and clean data to support AI/ML models and analytics.
What you’ll bring
- Maintain high-throughput streaming pipelines ingesting logs from firewalls, cloud, endpoints.
- Write parsers to normalize raw logs into standard schemas like OCSF or ECS.
- Optimize cost by routing high-value data to SIEM and bulk data to object storage.
- Clean and structure data for AI/ML detection models and advanced analytics.
Nice to have
- Experience with vector databases for storing embeddings.
- Knowledge of log observability/routing middleware tools.
- Familiarity with big data frameworks such as Spark or Flink.
Skills
Python for ETL and SQL for complex queries.Kafka or Pub/Sub message queues and stream processing concepts.Regex and log parsing for Syslog, CEF, JSON formats.Data lake principles using Parquet/Avro and data warehouse concepts.