AddressResponsibilities :
1. Design and Develop Spark Applications:
- Understand the current Apache Spark platform code and nuances related to the same
- Optimize and fine-tune Spark jobs for performance and efficiency.
2. Python Programming:
- Utilize Python for scripting, data manipulation, and integration with Spark applications.
- Collaborate with cross-functional teams to integrate Python-based solutions into Spark workflows.
3. Big Data Processing:
- Work with large-scale distributed systems for processing and analysing big data.
- Implement best practices for data processing, storage, and retrieval.
4. Performance Optimization:
- Identify bottlenecks and optimize Spark applications for enhanced performance.
- Implement caching and partitioning strategies to improve data processing speed.
5. Collaboration and Communication:
- Collaborate with data scientists, analysts, and other engineering teams to understand business requirements and translate them into technical solutions.
- Communicate effectively with team members and stakeholders.
6. Monitoring and Troubleshooting:
- Implement monitoring solutions for Spark jobs and proactively address any issues.
- Troubleshoot and resolve performance and operational issues.
7. Documentation:
- Maintain comprehensive documentation for Spark applications, workflows, and processes.
- Provide training and knowledge transfer to junior team Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
- 5-7 years of experience working with Apache Spark.
- Familiarity with AWS Cloud platform
- Strong proficiency in Python programming.
- Experience with big data processing frameworks and distributed computing.
- Solid understanding of data structures, algorithms, and data modeling.
- Excellent problem-solving and analytical skills.
- Effective communication and collaboration skills
