Senior Data Engineer

We have an immediate contract requirement for Senior Data Engineer with one of our direct clients in USA. Contract and Remote job (Need to support during EST business hours).

About the Job duties

• Construct data pipelines using Airflow and Cloud Functions to meet business

requirements set on from the Product and Engineering teams

Maintain and optimize table schemas, views, and queries in our data warehouse and

databases

Perform ad-hoc analysis to troubleshoot stakeholder issues surrounding data and

provide insights into feature usage

Develop workflows and integrations with data partners to expand our data capabilities

Document data architecture and integration efforts to provide a clear understanding of

the data platform to other team members

Provide guidance on data best practices when building out new product lines

Automate long running manual processes to free up internal resources

Create scalable framework for onboarding new data partners

Must have

Experience with data task orchestration (Airflow, CRON, Prefect etc..) with dependency

mapping

Data analysis and data modeling

Strong experience in Python, SQL, shell scripting

Experience interacting with APIs, SFTP, Cloud Storage locations (eg. GCS, s3)

Analytical problem solving skills, ability to analyze application logs to troubleshoot issues

Familiarity with Cloud Computing (GCP a plus)

Nice to have

Experience with Javascript

Hands on work with Airflow

ML / MLOps

Docker and K8s

NoSQL

Exposure to producer / consumer messaging systems

Some additional notes from our consultant onsite

Essential skills :

SQL

Python

Git

Apache Airflow (or other orchestrators such as Dagster or Prefect)

Nice-to-have skills :

Node.js or JavaScript

Google BigQuery

Event-driven serverless cloud functions (Google Cloud Functions, AWS Lambda or Azure Functions)

Asynchronous event-driven messaging (Google Cloud Pub / Sub, AWS SNS, AWS SQS, Azure Service Bus, Azure Event Grid or Azure Storage Queues)

Object storage : either cloud-based (Google Cloud Storage, AWS S3 or Azure Blob Storage) or on-premises (e.g., MinIO, Ceph)

Bachelor's degree required.