Skip to content
This repository has been archived by the owner on Jan 18, 2024. It is now read-only.

Waze WARP takes your CCP data feed and processes it into your cloud provider for access, analysis, and visualization. An Open Government Coalition (OGC) project. @GovInTheOpen

License

Notifications You must be signed in to change notification settings

LouisvilleMetro/WazeCCPProcessor

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Waze WARP

A free, open source cloud data processor for mobility use cases.

WARP (Waze Analytics Relational-database Platform) takes the Waze CCP data feed and processes it into a cloud database system for historic and real-time querying, analysis, API hooks, mapping, and integrations.

Overview

Louisville has created an automated cloud processing solution that can be replicated by any CCP Partner in the cloud, with the help of other govs, partners, and sponsors.

You grab this project's Terraform.io code and deploy the infrastructure-as-code stack (currently AWS but cloud agnostic).

You enter your CCP data feed URL as a parameter.

Then you can store, analyze, query, extract live and historic data for your city for many use cases.

See the Projects area for how you can help, and the Wiki for all the details.

Premier Project Sponsors

Organizations that are helping to fund, manage, collaborate, promote, and support the project.

Sponsors

See the project wiki for more details.

Deploy the Solution to Your Cloud

We have an end-to-end data processor, database, and analytics tools working that you can deploy. It saves your CCP data as JSON files every 2 minutes, and processes the data into a combined real-time and historic database.

See /docs/technical-deployment.md for very detailed instructions on how to get up and running in 30 minutes.

Finished Result

This creates an infrastructure stack which pings your custom Waze CCP data feed every 2 minutes and saves the JSON to a new bucket, which then gets processed into the relational database. There is error handling and also notification options for when things go right or wrong.

Here's what is created:

Waze Current Architecture

You can update the stack with new infrastructure as the code here gets updated, and it only affects new and changed items. You can also remove all the infrastructure automatically (minus the S3 bucket you created manually) by deleting the Terraform stack using terraform destroy after the get and init commands.

Building on the Platform

Since this is managed like software product, tools and integrations can be built upon it, and used by anyone who has deployed it. Examples include:

Create your own integrations and see our Issues for ideas.

Loading Historic JSON Data Files

You can also dump any previously collected historic JSON files into your bucket and the processor will go through them and save/update the relevant data into your database. Using aws s3 cp is a good place to start to copy files in chunks from a previous bucket to a new bucket.

See /docs/technical-backloading-data.md for very detailed instructions!

Costs

This config stands up infrastructure that is pretty inexpensive (and is about the same for any sized city). The database itself is the main cost (95+%) and will result in monthly charges under $200. We are working on ways to reduce the costs and you can help out on this issue.

Current Plans

We are working on writing API hooks, data visualizations and tools, and maps, which is all part of our project roadmap.

See our Projects and Releases areas for our blueprint of how we are proceeding.

We would like to collaborate with you! Please suggest updates, work on the help wanted issues, collaborate on the Wiki, etc. It would be great to work together to get the best solution, use cases, and finish faster.

We've build out the code in Terraform and supported AWS at first, but would like it to be deployed to any cloud provider. See our Issues area for how you can help with this.

Background

If you'd like a little more background on Louisville and what our city has been going with Waze and other mobility data, take a look at these links:

  1. Louisville Waze Internal Hackathon Recap
  2. Harvard Civic Analytics Network Presentation - Slides
  3. Waze April 2018 Monthly Call - Slides
  4. Open Government Coalition Supporting Framework
  5. Run Free Traffic Studies Using This Tool
  6. Traffic Study Tool Version 2 with Intersection Analysis
  7. UPenn Masters in Urban Spatial Analytics Program Collision Prediction Model