Job description
We are a digitally native company that helps organizations reinvent themselves and unleash their potential. We are the place where innovation, design and engineering meet scale. Globant is 20 years old, NYSE listed public organization with more than 27,000+ employees worldwide working out of 25 countries globally. Experience: 4 - 12 Years Job Location : Pune / Indore /Ahmedabad / Hyderabad Job Description We are seeking a highly skilled Azure Data Engineer to join the Data Engineering team and play a crucial role in an optimization initiative. The current problem is that read speed is compromised and faster responses from delta are required. The ideal candidate should have extensive experience in Python programming and be proficient in PySpark and Databricks. The selected candidate will be responsible for designing and implementing efficient data pipelines and solutions for data-driven projects in Azure Data Factory and Azure Databricks Responsibilities: - Design, build, and maintain scalable data pipelines using PySpark and Databricks
- Optimize data processing and storage for maximum performance and efficiency
- Troubleshoot and debug data-related issues, and implement solutions to prevent reoccurrence
- Collaborate with data scientists, software engineers, and other stakeholders to ensure that data solutions are aligned with business goals
Requirements: - Strong experience in Python programming and PySpark, and SparkSQL
- Clear understanding of Spark Data structures, RDD, Dataframe, dataset
- Expertise in Databricks and ADLS
- Expertise handling data type, from dictionaries, lists, tuples, sets, arrays, pandas dataframes, and spark dataframes
- Expertise working with complex data types such as, structs, and JSON strings.
- Clear understanding of Spark Broadcast, Repartition, Bloom index filters
- Experience with ADLS optimization, partitioning, shuffling and shrinking
- Ideal experience with disk caching
- Ideal Experience with cost based optimizer
- Experience with data modeling, data warehousing, data-lake, delta-lake and ETL/ELT processes in ADF
- Strong analytical and problem-solving skills
- Excellent documentation, communication and collaboration skills
Job Segment: Database, Data Modeler, Data Warehouse, Engineer, Product Development, Technology, Data, Engineering, Research Refer code: 929845. Globant - The previous day - 2024-02-20 14:33
Address
Form of work
