REMOTE Databricks Data Engineer - Insight Global : Job Details

Job Description

We are seeking a skilled and motivated Data Engineer to join our Provider 360 Informatics Data Team, focused on building a robust data lake and GraphQL API infrastructure for provider data in the healthcare industry. This person will join a team of 4 engineers and will collaborate closely with their team members on daily tasks and projects. The Provider 360 (P360) team focuses on aggregating and managing provider, clinic, and hospital data to support critical business functions such as helping patients find providers with specific specialties or from certain geographic areas. This role is open to support a provider data consolidation and enhancement initiative. The team is building out comprehensive provider data pipelines and improving data accessibility for member-facing applications and internal reporting systems. They need a data Engineer to come in and work on the Provider 360 initiative, focusing on provider data aggregation, transformation, and delivery. The role involves taking provider, clinic, and hospital data from various sources and building out complex data structures that enable patients to search for providers by specialty, location, and other key criteria. Some of their responsibilities will include reviewing existing data processes, implementing CI/CD pipelines using Terraform, integrating with various APIs, and optimizing data storage solutions using CosmosDB and PostgreSQL.

Key Responsibilities

Design, build, and maintain scalable data pipelines for ingesting, cleaning, and transforming provider data.

Develop and optimize workflows in Databricks for large-scale data processing.

Implement and manage data storage solutions using the Microsoft Azure suite, including Azure Data Lake, Blob Storage, and Azure SQL.

Collaborate with API developers and data consumers to ensure seamless API data consumption.

Work closely with data scientists, analysts, and product owners to ensure data quality, consistency, and availability.

Contribute to the evolution of our data lake and warehouse architecture to support current and future analytics needs.

Technical Environment

Cloud Infrastructure: Terraform for infrastructure as code

Version Control and Collaboration: GitHub

CI/CD: Terraform-based deployment pipelines

Databases: CosmosDB for NoSQL data management, PostgreSQL for relational data storage

API Integration: RESTful services and data ingestion

Data Processing: ETL/ELT pipelines

Provider Data: Aggregated provider, clinic, and hospital datasets

We are a company committed to creating inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity employer that believes everyone matters. Qualified candidates will receive consideration for employment opportunities without regard to race, religion, sex, age, marital status, national origin, sexual orientation, citizenship status, disability, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to Human Resources Request Form ( . The EEOC Know Your Rights Poster is available here ( .

To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: .

Skills and Requirements

Hands-on experience with Databricks and Apache Spark.

Proficient in SQL, Python, PySpark, Git, PowerBI

Strong proficiency with Microsoft Azure cloud services, especially in data storage and compute

Proven experience with data lakes and/or data warehouses, understanding data warehousing concepts and dimensional modeling

Solid understanding of REST APIs and experience consuming them in data workflows; API Integration experience for data ingestion and service connectivity

Experience with data cleaning techniques, and ETL/ELT for data transformation and CI/CD pipeline development, including deployment with Terraform

Experience with CosmosDB or PostgreSQL databases Experience building or maintaining GraphQL APIs.

Experience designing and developing REST APIs.

Familiarity with AI/ML integration in data pipelines or analytics workflows.

Knowledge of healthcare data standards and provider data models. null

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].