project.md

Project

Goals

As stated in the course description:

Over the semester, students will build a complex end-to-end data system.

You'll be building a live dashboard, with all the infrastructure behind it:

• Automated data ingestion
• A database
• Web-based interactive data visualization

Data flow:

API -> Python script -> BigQuery -> Streamlit app (on the internet) -> visualization

All of this will be in the cloud.

Inspiration

• Center for Disease Control (CDC) dashboards
• Chicago Region Transit Dashboard
• Chicago Transit Authority Historical Bus Crowding
• Colorado Behavioral Health Administration (BHA) Performance Hub
• Congestion Pricing Tracker
• Darkhorse Visualizations
• Gapminder
• Inside Airbnb
• MTA
  • Capital Program Dashboard
  • Metrics
• New York Flu Tracker
• New York Traffic Data Viewer (TDV)
• NYC Population FactFinder
• NYPD TrafficStat
• Tax Wrapped
• TransitMatters
• United States of Health

Expectations

At all times

• Pair program as much as possible.
• All code is peer-reviewed through pull requests.
• All team members are contributing equal amounts.
• The Project leverages at least one dataset that's regularly updated.
• The code, documentation, and repository are clean, following good coding style and other best practices.

By the end

• Create dashboard(s) that explore relationships between variables.
  • It should be more open-ended than Projects from Computing in Context, allowing users to explore.
  • The site doesn't need to read like a blog post necessarily, but it should explain what's going on.
  • It should layer several sources/dimensions of data to add depth and context (e.g. location, weather, demographics).
  • It should have multiple visualizations that work together.
  • The idea is that it should make it easy for someone (maybe a policymaker or other decision maker) to spot meaningful patterns or insights.
• The site + codebase should be a polished portfolio piece.
• Data is being automatically updated.

Teams

See the grouping by team_id. The teams have a corresponding repository and Google Cloud Project automatically created.

Part 1

Goals

Your group will pick an initial:

• Problem space
• Dataset

Part of this project is getting experience with automated data ingestion. Doing so is more interesting with data that changes regularly. You can incorporate additional datasets in the future.

Steps

Do the following as a group:

1. Discuss what you'd like your project to focus on.
   • Don't need to get too specific yet.
   • This doesn't lock you in.
2. Explore datasets that are updated weekly (the more often, the better) and pick one that could be relevant.
   • You can pick multiple, but start simple.
3. Create a new notebook in your team repository.
4. In the notebook, load and display the data from all your data sources.
   • If there's an API, use the API.
5. Draw one (or more) example visualizations that you'd like to produce.
   • Have at least one be a time series.
   • You can do so digitally or on a piece of paper.
   • Include a title, legend, and axes labels (where appropriate).
   • This is just a sketch; don't worry about the specific values.

Proposal

You will then submit the following to the Discussion on Ed:

• What dataset are you going to use?
  • Please include a link.
• What are your research question(s) What questions should your users be able to answer by using your app?
  • Come up with at least three.
  • Go deep, not broad.
• What's the link to your notebook?
• What's your target visualization?
  • Include a picture.
• What are your known unknowns?
• What challenges do you anticipate?

Only one person from your group needs to submit. None of this is set in stone long term, it is just a starting place. It can all be changed later.

Lab 3

Part 2

Goal: Get experience with an application development framework

Steps

1. Using your dataset from Part 1:

   1. Create a Streamlit app.
   2. Deploy to the Streamlit Community Cloud.
   3. Add a visualization.
      • You can get fancy, but don't have to at this stage. Get something simple working first.

2. Bring in a second relevant dataset. (This one doesn't need to be regularly updated.)

   • This can be shown on a separate page of your Streamlit app, or combined in a single visualization.

3. Add the names of the people on your team to your Streamlit app homepage.

4. Set the repository Website to the app URL (https://<something>.streamlit.app/).

   

5. Turn in the link to your live app via CourseWorks.

Tips

• You can load data:
  • from a URL (preferred), either:
    • An API
    • A link to a CSV
  • from a file, checked into the repository
• Note the Streamlit app resource limits.
• At this stage, feel free to make the dataset small to get it working.
• If the app is slow to reload, experiment with caching.

Part 3

Goal: Get experience with unit testing

Steps

Work on branches and submit pull requests for the chunks of work — you decide what the "chunks" are.

1. Without writing any code:
   1. Review your existing code.
      • What can be refactored into functions?
      • Where can we make our code DRY?
   2. Decide what function you're going to create.
   3. Come up with test cases (inputs) and expected outputs.
      • This can be in a text file, doc, piece of paper, etc.
2. Then, as code:
   1. Write tests.
   2. Confirm they fail.
   3. Refactor your code into the function.
   4. Make the tests pass.
3. Repeat until you feel your code is well-organized and well-tested.
4. Submit the links to the pull requests via CourseWorks.

Outcome

As a result, your:

• "main" scripts (for Streamlit pages or otherwise)
• Functions

should be relatively short and easy to read.

This isn't a one-time thing; continue testing and refactoring as you continue with the Project.

Lab 6

Part 4

Retro

You will hold a team retrospective, with the goal of improving how your team works together. Since the groups are small, it can be fairly informal.

1. Schedule 45 minutes for the retro.
   • The retro needs to be done live/synchronous, not asynchronous.
2. Read about retros.
3. Decide who will be the Facilitator.
   • Optional: Get someone from outside the team.
4. Facilitator: Set up EasyRetro. Instructions.
5. In the actual retro:
   1. Read the Agile Prime Directive out loud.
   2. 5 minutes: Individually write down "what went well" and "what could be better".
   3. 10-15 minutes: Discuss what has gone well.
   4. 20-25 minutes: Discuss what could be better.
   5. 5 minutes: Document takeaways / action items.

Keep going

1. Move your Proposal to the Streamlit app as is.
   • How to add text elements
2. Revisit the Proposal.
   • Any new insights?
   • Anything you want to adjust?
3. Document any changes to the Proposal on the Streamlit page.
4. Proceed with enhancing the app.
   • If the majority of your code (to call APIs, etc.) is in modules/functions, it can be imported from a Jupyter notebook. You can do exploratory analysis there, moving things to modules/Streamlit as you go.
   • You might not be able to fully answer the question(s) yet, but get as close as you can.

At this point, your project should be looking more like one of the examples. Looking through the Streamlit data elements may be helpful.

Submit

Submit links to:

• The EasyRetro board
• Jupyter notebook(s), if any
• The (updated) Streamlit app

Part 5

Goal: Understand how to work with a cloud-based database

Notes

• A service account has been created in your Project for you. It has been given read-only access to BigQuery.
• There are various things that can go wrong in these steps. Don't wait until the last minute.

Steps

Do the following for your regularly-updated data source. Only do one for now — we'll do the rest in Lab 10.

1. Install pandas-gbq.
2. Load data.
   • Create a Python script that:
     1. Creates the table, if it doesn't exist
     2. Pulls data from your data source
     3. Copies the data to BigQuery using the appropriate technique
   • Since you'll be running the script locally, authenticate with a user account.
   • How to write tables with pandas-gbq
   • How will you know it worked as intended?
3. Have your app use BigQuery.
   1. Each team member will need to:
      1. Create a service account key as JSON. The service account is streamlit@[project].iam.gserviceaccount.com.
      2. Set up secrets management locally.
         • Make sure to add secrets.toml to your .gitignore so that you don't accidentally commit it to Git.
      3. Copy the key information to your secrets.toml file.
   2. Modify your app to read data from BigQuery.
      • Simplified example
   3. Copy the secrets to your deployed app.
   4. Re-deploy.
4. Submit the links via CourseWorks for:
   • The pull request(s)
   • The link to your live Streamlit app

Check-in

• We'll do this once now, once at the end.
• This will be factored into an individual score.

Lab 10

Lab 11

Part 6

Goal: Your regularly-updated data is being pulled in automatically.

Make a workflow that runs on a schedule to run your ETL. Check out the workflows in this repository as examples you can start with.

Thinking ahead

• How will you know if it worked?
• If something went wrong, where would you look to find out why?

Hints

Example workflow and code.

There's an extra step

GitHub Actions will need credentials + permissions to write to BigQuery.

How to do it

1. Create a service account in Google Cloud.
2. Grant it the appropriate role.
3. Create a key as JSON.
4. Add it to GitHub Actions as a secret.

Tips

• To test out the workflow, you can:
  • Make the schedule more frequent
  • Set it to push
  • Run it manually
  • Advanced: Run it locally
• To protect yourself in case your data gets messed up:
  • Set up table snapshots.
  • Make separate test and production datasets/tables in BigQuery.

Submit

Submit links via CourseWorks for:

• Your pull request(s)
• A successful run of the GitHub Action

Part 7

Goal: Determine and prioritize TODOs

You'll do this prioritization exercise as a group.

• This must be done synchronously.
• Look back at the expectations.
• The Prep can be done in the meeting itself.
• You can use paper/stickies or a digital template like Miro's

Submit a photo/link to the matrix via CourseWorks.

Part 8

Refinement

Goal: Meet the expectations

Do tasks you came up with in the prioritization exercise in order of priority.

Presentation

Goal: Force clarity of the project and code by having to show and explain them to others

Each group will do a presentation on their Project in class.

• 10-ish minutes to present, 3-ish minutes for Q&A
• Slides optional
• Everyone in the group should speak.
• Explain the initial proposal and how it's evolved.
• Show the live app.
• Walk through the code.
  • You're giving a tour of what's unique about your system.
  • You don't need to explain every line.
  • For example, you might say things like:
    • "We have our files organized as …"
    • "In this function, we're …"
    • "To improve performance, we changed this part to …"
• Talk through what you learned in doing the Project.

Peer feedback

Students will fill out the feedback form for all the presentations they observe.

Personal writeup

Goal: Narratively document your Project

While the presentation shows your Project to the other students, we also want a writeup that you can show off to potential employers, etc.

• Each team member: Write a few paragraphs.
  • This can be (in order of preference):
    1. A page on the portfolio site you set up in Computing in Context
    2. A page on a separate personal site
    3. A LinkedIn post
    4. A Google Doc
  • The content can be very similar to the presentation.
  • While each team member's writeup will be similar, it should be in your own words.
  • You're welcome to write in whatever language you prefer.
  • Include screenshots of the live app, in case the live one goes down.
• Link to those blog posts from the Project site/README.

Submit a link to the page via CourseWorks.

Final check-in

Goal: Give the instructors a sense of how the teamwork went