Sr. Data Engineer

PFB the JD for the Sr. Data Engineer role for a Product based organization :Role And ResponsibilityDesign and develop robust, fault-tolerant, and scalable data platforms and pipelines integral to empowering organization's data products and AI/ML applications.Collaborate closely with internal engineering teams and partners to decipher requirements, formulate functional specifications, and successfully deploy solutions intoproduction.Embrace a robust product-oriented mindset, ensuring that code aligns with core business needs and translates into impactful outcomes.Display an unwavering commitment to ongoing learning and adaptation, readily embracing new technologies and frameworks as necessary to fulfill project demands.Perform comprehensive code reviews for fellow team members, advocating for the implementation of clean, secure, and unit-tested coding best practices.Address intricate engineering challenges head-on, proposing innovative solutions that contribute to the advancement of our technology landscape.Exhibit proactive initiative in identifying gaps and furnishing effective solutions that drive continual improvement.Skills Required5+ years of experience as a backend engineer in building enterprise-grade software products with at least 4 years of experience in building big data products.Proficiency in Python programming, along with a commanding grasp of SQL and Pandas for effective data manipulation.Ability to work with both batch and real-time data processing pipelines.Experience with big data tools and technologies like Apache Spark, Hadoop, Hive, Kafka, and Kinesis.Hands-on experience orchestrating big data pipelines using tools like Apache Airflow, AWS Step Functions, and equivalent solutions.Comprehensive understanding and practical knowledge of at least one major big data storage platform, including but not limited to Delta Lake, Apache Hudi, or Iceberg.Professional experience working with at least 2-3 of the core AWS big data technologies like Glue, EMR, QuickSight, Athena, S3, Lambda, Redshift, Kinesis etc. Experience insimilar technologies offered by a competing cloud vendor or open source is alsoacceptable.Hands-on experience with cloud data warehousing, building ingestion pipelines, data cataloging, and data lakes.Understanding of database design, schema development, and ETL processes.Experience with building and integrating REST-based APIs needed to integrate data sources.Experience in working with CICD pipelines composed of Git, Docker, Terraform, or equivalent.Understanding of building robust multi-tenant backend products with clear separation of concerns to ensure data integrity, security, and privacy.Understanding and experience in building production-quality software systems following design principles of Object Oriented Programming and best practices of DRY/KISS/YAGNI.Strong concepts in Data Structures and Algorithms. Nice To HaveExperience in data cataloging and data governance technologies like AWS Glue catalog, AWS Lake Formation, or equivalent.Exposure to building data mesh or data fabric architectures at enterprise scale.Understanding and exposure to working on Vector databases like Pinecone and Graph databases like AWS Neptune for developing data pipelines for AI/ML products.Exposure to MLOps and working on ML products backend data pipelines using AWS Sagemaker/Bedrock or equivalent.Experience in working in the cybersecurity industry, and understanding the basics of the current security landscape (attack frameworks, security log processing, basic knowledgeof AV/EDR/DLP/CASB, etc.) is a plus.Skills: athena,awssagemaker,cybersecurity industry,aws step functions,security log processing,redshift,iceberg,dlp,cicd pipelines,yagni,aws,kinesis,multi-tenant backend products,terraform,kiss,apache hudi,rest-based apis,quicksight,aws lake formation,aws glue,apache spark,docker,aws glue catalog,apache airflow,database design,s3,lambda,cloud data warehousing,python,casb,etl processes,data structures,hadoop,delta lake,emr,mlops,ci/cd,bedrock,edr,git,vector databases,graph databases,dry,algorithms,av,kafka,object oriented programming,big data,pandas,hive,python programming,sql,etl