Data Engineer
: Job Details :

Role Description: Job Overview: We are seeking a talented and motivated Data Engineer with expertise in Spark SQL, Databricks, Azure Data Factory (ADF), SQL, IICS, Unix, PySpark, Python, and Azure Data Lake Storage (ADLS) to join our dynamic team. In this role, you will be responsible for designing, developing, and maintaining scalable data pipelines, optimizing data workflows, and ensuring efficient data processing. You will collaborate closely with data scientists, analysts, and other stakeholders to ensure the timely delivery of data solutions.Key Responsibilities: Design, develop, and maintain ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, PySpark, and Python. Build and optimize Spark SQL queries and workflows to process large volumes of data. Implement and manage data pipelines and data integration solutions in cloud environments, specifically on Azure. Leverage PySpark and Python to develop scalable, distributed data processing solutions and build custom data transformations. Work with IICS (Informatica Intelligent Cloud Services) for data integration, transformation, and management. Develop and optimize data workflows to support analytics and reporting needs. Write complex SQL queries to extract, manipulate, and load data from various sources into data warehouses. Utilize Unix/Linux shell scripting for automating tasks, managing file systems, and processing data.Work with Azure Data Lake Storage (ADLS) to store, manage, and access large-scale data sets in a cloud environment. Collaborate with data scientists, analysts, and other stakeholders to ensure the data infrastructure meets business requirements. Ensure data quality, consistency, and security through best practices and governance frameworks. Troubleshoot and resolve data processing issues, ensuring the scalability and reliability of data pipelines. Monitor and optimize the performance of data solutions for efficiency and cost-effectiveness. Competencies: Digital : Databricks, Azure Data Factory Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.