Data Engineering and Manipulation
You can work with other technologists and analysts to integrate and separate data feeds in order to map, produce, transform and test new scalable data products that meet user needs. You have a demonstrable understanding of how to expose data from systems (for example, through APIs), link data from multiple systems and deliver streaming services. You know how to work with other technologists and analysts to understand and make use of different types of data models. You understand and can make use of different data engineering tools for repeatable data processing; you can compare between different data models. You know how to build scalable machine learning pipelines and can combine feature engineering with optimisation methods to improve the data product performance.
Level 1
You understand the technical principles underpinning data feeds (both streaming and batch types) for consumption by data products. You can explain how existing tools can be applied to create these feeds. You know about and can use ad hoc data exploration techniques to build datasets for consumers. (Relevant skill level: awareness)
Level 2
You can work with data engineers to map, produce, transform and test new data feeds for data owners and consumers, using tools and technologies already in use in the business area. You can conduct ad hoc data exploration in common data serialisation and storage formats used across the business for consumers. (Relevant skill level: working)
Level 3
You can work with data engineers to map, produce, transform and test new data feeds for data owners and consumers, selecting the most appropriate tools and technologies. You can lead ad hoc data exploration in a wide variety of data serialisation and storage formats, from across the business, for consumers. (Relevant skill level: practitioner)
Level 4
You have significant experience building data pipelines at scale using a range of tools and technologies; you can advise on data engineering best practices across industry. You can lead ad hoc data exploration practices in your business area by identifying and setting best practice and standards. You know how to identify opportunities for future innovation of data exploration practices. (Relevant skill level: expert)