Provide technology ownership for data solutions for projects that the team has been tasked with.
Work with a cross functional team of business analysts, architects, engineers, data analysts and data scientists to formulate both business and technical requirements.
Design and build data pipelines from various data sources to a target data warehouse using batch data load strategies utilizing cutting edge cloud technologies.
Conceptualizing and generating infrastructure that allows data to be accessed and analyzed effectively.
Documenting database designs that include data models, metadata, ETL specifications and process flows for business data project integrations.
Perform periodic code reviews and test plans to ensure data quality and integrity.
Provide input into strategies as they drive the team forward with delivery of business value and technical acumen.
Execute on proof of concepts, where appropriate, to help improve our technical processes.
Conduct Technical assessments and mentor junior team members
Skills Required
7-10 years of Data Engineering experience
4+ years of experience working on Spark (RDDs / Data Frames / Dataset API) using Scala/Python to build and maintain complex ETL pipelines.
4+ years of experience in the big data space
Experience in translating business requirements into technical data solutions on a large scale.
2+ years of experience working on AWS (Kinesis / Kafka / S3 / RedShift) or Azure.
Able to research and troubleshoot potential issues presented by stakeholders within the data ecosystem.
Experience with GitHub and CI/CD processes
Experience with Compute technologies like EMR and Databricks
Experience working job orchestration (eg., Airflow / AWS Step Function)
Experience with Data Modeling, Data warehousing
Working with Kubernetes is a plus
Strong analytical and interpersonal skills.
Enthusiastic, highly motivated and ability to learn quickly.
Able to work through ambiguity in a fast-paced, dynamically changing business environment.
Ability to manage multiple tasks at the same time with minimal supervision.
