AddressPosition: Senior Kafka Architect
Location: Bangaluru
Duration: Full Time
MUST HAVE SKILLS:
A: Kafka
B: Storm
C: Architectural components: brokers, Zookeeper, Producers/Consumers
JOB DESCRIPTION:
- Looking for a senior Kafka engineer/architect with an established track record in Kafka technology, with hands-on production experience and a deep understanding of the Kafka architecture and internals of how it works, along with interplay of architectural components: brokers, Zookeeper, Producers/Consumers etc.
SOME OF THE KEY RESPONSIBILITIES ARE LISTED BELOW:
- Architecture Design: Design Kafka-based data streaming architecture that meet scalability, fault-tolerance, and performance requirements.
- Cluster Sizing: Determine the appropriate cluster size, partitioning, and replication factors based on workload and data volume.
- Topic Modeling: Define Kafka topic schemas, naming conventions, and access control policies for efficient data organization.
- Security Planning: Implement security measures like SSL/TLS encryption, SASL authentication, and ACLs to protect data and Kafka infrastructure.
- High Availability: Ensure Kafka clusters are highly available and fault-tolerant by configuring replication, leader election, and monitoring.
- Disaster Recovery: Plan and implement disaster recovery strategies to ensure data integrity in case of cluster failures.
- Performance Optimization: Fine-tune Kafka configurations, monitor performance metrics, and optimize resource utilization for efficient data processing.
- Monitoring and Alerting: Set up monitoring tools and alerting systems to detect issues, track performance, and respond to anomalies.
- Load Testing: Conduct load testing and benchmarking to validate the Kafka architecture’s performance and scalability.
- Cost Management: Optimize resource usage to control operational costs while maintaining performance and reliability.
- Training and Support: Provide training to other team members and support staff on Kafka best practices and troubleshooting.
- Documentation: Maintain comprehensive documentation for Kafka configurations, data flows, and architecture diagrams.
- Scaling Strategies: Develop strategies for horizontally and vertically scaling Kafka clusters based on changing workloads.
- Hands-On Experience: Hands-on experience as a developer who has used the Kafka API to build producer and consumer applications/components. Be able to come up with library/framework for common functionality/use cases to have different teams reuse code rather than writing new code.
- Message Formats: Strong familiarity of message formats such as XML, JSON, CSV, etc. along with serialization/deserialization options.
