Data Platform Engineer

About the Role and Reponsabilities :

In this role you will have the chance to build large additions to our platform ecosystem, contributing to an infrastructure that centralizes our ETL and streaming infrastructure, enables experimentation and data development, and powers algorithm development. You will design and build scalable engineering solutions to solve business and data problems.

Core tech stack used by this role :

• S3 using Parquet and Iceberg for data storage
• Hive and Iceberg REST catalog as our metadata store
• Spark (Python, SQL, Scala) for data transformation
• Go, Scala and Java for backend development
• Kafka for data streaming
• Kubernetes and Helm for service deployments
• AWS IAM for access control
• Datadog for logging alerting and monitoring
• Trino as our federated query engine
• Identify, develop, and evangelize tooling, approaches, and data products that raise the bar of validity in experiment design and analyses across the company
• Collaborate with multiple teams and functions to deliver deep-dive analyses and experiment design for new initiatives and high-priority business areas
• Develop and iterate on our processes to empower more efficient and rigorous experimentation

Requirements

Avenue Code reinforces its commitment to privacy and to all the principles guaranteed by the most accurate global data protection laws, such as GDPR, LGPD, CCPA and CPRA. The Candidate data shared with Avenue Code will be kept confidential and will not be transmitted to disinterested third parties, nor will it be used for purposes other than the application for open positions. As a Consultancy company, Avenue Code may share your information with its clients and other Companies from the CompassUol Group to which Avenue Code’s consultants are allocated to perform its services.