Code.
Welcome back to Open Data Stories Ottawa. I’m sorry for the long delay in posts. Partly, I’ve just been feeling really lazy lately. My work let me start using Python in my job, and I’ve been using up a lot of my learning energy there. Side note, I definitely feel a lot more comfortable using Python since I’ve had that opportunity.
Second, I was accepted to a Master’s of Data Science at Ryerson! Yay me.
Of course, it’s not that I haven’t been doing anything. It’s just that my pace has been a lot slower, and the work has been a little different.
So, what have I been up to, you ask? (I’m kidding, I know you weren’t asking. But I’ll tell you anyways).
This.
Cool, you put an external hard drive on the floor.
Ah my friend, but this is not just any hard drive. Actually, I’ve hooked it up to a Raspberry Pi, and I’m currently collecting data. All day, every day.
Can anyone guess what this is?
Ok, you’re probably tired. Promise this is the last guess.
That is a map of Ottawa’s transitway, depicting all the stops.
That’s right, the data I’ve been collecting every day is from OC Transpo.
Building the tool was actually a really interesting learning experience. Like being the programmer and the IT guy, rolled into one. If you thought those were the same person, I can now attest, you are wrong.
Without further ado, allow me to present to you my transit data collection tool.
My idea for this started with an observation. I was waiting for the bus at Tunney’s Pasture, as I do most days after work. Once again, east-bound busses were in short supply. Meanwhile, I watched probably thirty-five busses go the other way.
Rather than balling up my frustration and calling OC Transpo, or worse, writing a plaintive letter to the Citizen, I got to thinking. What if I made use of that OC Transpo API I made an account for and never did anything with three years ago.
Eureka!
At first, I thought about writing a simple Python application and just collecting data with my personal computer. But then I might put my computer in the ground faster.
Next, I thought about building that same application, but hosting it on a cloud service. I actually made an attempt, but realized I really didn’t have the knowledge yet, and I did not want this to take forever.
So the last thing I settled on was spending my money on the physical infrastructure. One trip to Amazon and I had myself a shiny new Raspberry Pi 3.
I actually received an external hard drive for my birthday (shoutout my parents), and we were rolling.
I wrote a python application that could ping the OC Transpo API roughly every 8 seconds. I connected the returning data to a MySQL database. And then I started it.
In the span of about a month, I’m up to over 11 million rows of data. Most of it is will be cut down prior to analysis, but it is quite a wealth of information. In the coming weeks, before I go to school, I’m going to try to do some analysis on the data, and build out that dashboard a little more. I just moved, but I’m settling in and should have a bit of time before I have to start studying. Talk to you soon!