Data Engineer Remote role - Fromjimmy : Job Details

We are looking for Data Engineers for the data projects of our Fortune50 client. The main objective is to modernize the client\'s data infrastructure by migrating to Databricks. The solution is a cloud-based platform designed to empower users to build datasets for their projects in a swift, efficient, and standardized manner. The plan is for the data to power the GenAI in the future.

This is a

for engineers based in Europe, Turkey, and Middle East with a required overlap of

• Migration of data infrastructure to Databricks

• Transition to DBXaaS

• Design data models

• Develop SQL code to define data structures and transform data from staging to marts

• Create Source to Target Mappings (STMs) for ETL specifications.

• Evaluate data sources, assess quality, and determine the best integration approach

• Develop strategies for integrating data from various sources and data warehouses.

• Optimize and maintain the data pipeline for SDCM, ECM, and DCM, flattening JSON into Databricks tables.

• Work with data vault modeling (a plus).

• Implement changes to the JSON flattening process based on business needs.

• Write and execute unit tests to ensure code accuracy.

• Optimize the performance of the data pipeline and fix data quality issues

• Implement active monitoring for both data pipelines and data quality.

• Gather requirements, set targets, define interface specifications, and conduct design sessions

• Work closely with data consumers to ensure proper integration.

• Adapt and learn in a fast-paced project environment.

• Start DateASAP

• LocationRemote

• Working hoursUS time zone overlap required: 2-6pm CET

• Long-term contract based-role6+ months

• Strong Spark and SQL skills for ETL, data modeling, and performance tuning

• Experience with Databricks

• Proficiency in Python, especially for handling and flattening complex JSON structures

• Hands-on experience with Cloud architecture (Azure - preferred, AWS, GCP).

• Familiarity with Delta Lake and data warehousing concepts

• Experience with Azure Data Factory, Azure Cosmos DB, Azure functions, Event store, ADLS, Key Vault.

• Proficiency in RDBMS/NoSQL data stores and appropriate use cases.

• Understanding of software engineering and testing practices within an Agile environment

• Experience with Data as Code; version control, small and regular commits, unit tests, CI/CD, packaging, familiarity with containerization tools such as Docker (must have) and Kubernetes (plus).

• Excellent teamwork and communication skills

• Proficiency in English, with strong written and verbal communication skills.

• Efficient, high-performance data pipelines for real-time and batch data processing.

• Knowledge of cryptography and its application in blockchain is a plus.

• Experience with blockchain indexing is a plus.

• Familiarity with Delta Lake and data warehousing concepts is a plus.