Specialist - Data Engineering

Job Summary: Seeking a Specialist with 5 to 7 years of experience skilled in Scala and Apache Spark to design and maintain scalable big data solutions within the Hadoop and Spark ecosystemJob DescriptionDesign develop and maintain scalable data processing applications using Scala and Apache Spark Collaborate with crossfunctional teams to gather requirements and translate them into efficient big data solutions Implement data ingestion transformation and aggregation pipelines leveraging Spark and SparkSQL Ensure high performance reliability and scalability of data processing workflows Optimize existing data processing jobs for improved efficiency and resource utilization Stay updated with emerging technologies and best practices in big data and functional programmingRoles and ResponsibilitiesLead the development and deployment of complex big data applications using Apache Spark and Scala frameworksMentor junior team members and provide technical guidance on Scala and big data technologiesCollaborate with data engineers architects and analysts to design endtoend data solutionsTroubleshoot and resolve performance bottlenecks in distributed data processing jobsDevelop and maintain documentation related to architecture design and codeDrive innovation by evaluating and integrating new tools and frameworks within the Hadoop and Spark ecosystemEnsure adherence to coding standards security policies and compliance requirements