Senior Data Engineer - Hudson Valley Federal Credit Union : Job Details

Hudson Valley Credit Union is currently recruiting for the position of Senior Data Engineer. Primary Functions: Design, build, and optimize our data infrastructure, leveraging the power and scalability of Databricks within our Azure environment. Maintain a deep expertise in Databricks and its ecosystem in order to build high-performance data pipelines, ensure data quality, and enable advanced analytics and machine learning initiatives. Responsibilities Design, develop, and maintain robust and scalable data pipelines primarily using Databricks, leveraging Spark (Py. Spark or Scala), Delta Lake, and Databricks Workflows Build and optimize data lakehouse solutions on Azure Data Lake Storage Gen 2 integrated with Databricks Implement and enforce data quality frameworks and data governance policies within Databricks using tools and features like Delta Lake's ACID properties and data validation techniques Identify and resolve performance bottlenecks in Databricks-based data pipelines and Spa...Data Engineer, Engineer, Computer Science, Senior, Technology, Banking, Data