Data Pipelines with Snowflake and Streamlit

In this course, I construct a pipeline of data engineering that will gather multiple sources of information: both Kaggle datasets and Google Trends data fetched through SerpAPI. It is going to be a pipeline to aggregate and combine data on Netflix actors along with the trends about them on Google within weeks after a new show was released.

You will use Kaggle as a data source for your dataset regarding Netflix shows and actors and Google Trends, through SerpAPI, to fetch live search data from the actors. All this is going to be stored and processed within the Snowflake database, using its cloud-native architecture to maximize scalability and performance.

Technical Stack Overview:

Snowflake Database: Central repository for storing and querying data.

Streamlit within Snowflake: A web application framework where the data could be rendered directly within Snowflake

AWS S3: Where some intermediate dataset would be stored or fetched

Snowflake Python Procedures: The code for automating some parts of data manipulation or processing pipeline

Snowflake External Access & Storage Integrations: Manage secure access to storage of an external service.

By the end of this course, you will end up with a fully functional data pipeline that processes and merges streaming data, cloud storage, and APIs to enable trend analysis, visualized in an interactive Streamlit app within Snowflake.