The main objective of this project is to set-up a full-scale data pipeline that takes in consumed data from our businesses analytics dataset and then passes it forward into a set of specially designed features. In this task, the student will be helping our team in completion of this pipeline and the student will be specifically focused on a set of specific data engineering tasks. The finished product will be a design doc detailing all the inner workings of the pipeline, completed GitHub tickets, and a repo containing all the work done during this project.

We'll choose the tasks and schedule them appropriately for the allocated time:

• Connecting the CSV data to an SQL database
• Setting up custom filters and pre-processing data prior to entering the pipeline
• Setting up data analysis pipelines and connecting the pipeline to custom scripts checking for incorrect data
• Automating data labelling and correctly organizing data into appropriate locations
• Connecting ETL pipeline to REST API for internal usage
• Automating and deploying pipeline for continuous use

We'll be providing the student with complete mentorship and continuous support during this project. The student will be given full support while developing this project with dedicated mentorship hours, peer programming and live code review sessions. The student will also be provided access to all of our internal tools and access to our resources for the duration of this project.