Build and maintain data feeds
- As a Data Engineer, your primary responsibility is to build and maintain the data feeds that underpin the work of the Data Science and Insights team
- You will be responsible for the entire end-to-end of data provisioning, from identification and scoping through design, implementation, testing and ongoing management
- You will need to employ engineering best practices to create data feeds that are robust, scalable and efficient
Data Warehouse management
- The core of our data platform is a modern, scalable cloud data warehouse, from which we develop and deploy data products for use across the business
- You will play a leading role, alongside your IT colleagues, in the management and further development of this critical infrastructure
- In addition to the Data Warehouse itself, you will use various AWS services to deploy containerised data transformation applications and manage orchestration with Apache Airflow
Data stewardship
- The quality of our data assets is essential to their adoption and value generation. You will play a key role in ensuring data quality through the development of automated data validation and cleansing processes. Maintaining effective documentation is essential for usability of data assets. Your role will include documentation of data schemas, data lineage and transformation
Maintenance of deployed applications and reports
- Ensuring that necessary data is continually fed to deployed data science applications and reports
- Monitoring data flows, job orchestration and patches/releases as necessary
New applications and reports
- Assisting team members in provisioning data for applications and reports in development
- Optimising data processing and where appropriate, applying distributed methods