Exploring the MBTA’s realtime API – Part 1

I have been playing with Massachusetts Bay Transportation Authority (MBTA) Realtime API in recent weeks and ended up learning a lot about this transport agency which operates the largest fleet of public transport in New England area.
MBTA has a web based realtime API which can be used by developers who want to use the data to create new applications or just explore with the data.

Lots of the data provided comes with very little documentation in itself. Even when the documentation is provided it is pretty much lacking any context unless you read the blue book provided by MBTA. Here is a simple run-down of everything MBTA related, especially related to it’s subway system.

MBTA runs four rapid transit train lines, bus services, boats and commuter rails. All four of them are Electrical Multiple Units type of trains, with varying number of coaches per train. Out of four transit train lines, three of them are based on heavy rail system containing around 6 coaches per train with multiple doors on each coach. The Red Line train has 6 coaches, each coach contains 4 doors on each side of the coach. The Orange Line has 6 coaches and has 3 doors on each side of the coach. The Blue line has 6 coaches and has 2 doors on each side of the coach. The last line is Green Line which is actually a Street Car containing just two coaches and just one door on each side and might vary as there are multiple generation street cars serving the green line.

Blue line runs between Bowdoin in central Boston near Beacon Hill and ends at Wonderland in Revere. It is just one single line and does not split anywhere.

Red Line runs between Alewife at Somerville and runs south splitting near JFK/UMass where one section goes to Braintree and other goes to Ashmont.

The Orange Line runs between Oak Groke in North to Forest Hills in SouthWest.

The Green Line starts at Lechemere in Cambridge which is the Northern section and goes South till Boyslton Street, turns West and eventually splits into four sections are multiple stations. The four sections end at Boston College (B Line), Cleveland Circle (C Line), Riverside (D Line) and Heath Street (E Line). Even though there are four sections, not all trains run from each end to Lechemere. Some trains end way before the last stops on each section, not all trains from B, C, D or E Line go all the way upto Lechemere.

If you were confused by the above description, let me introduce you to the concept of “route”. A route is any two places where a service exists. Even in cases like Orange Line where there is no split of line, there can be theoretically more than one route. If there is a weekend service which travels from say Back Bay to Wellington station instead of going all the way till the ends of the line, that would count as another route on the same line.

We expect D line to go all the way from Riverside to Lechemere, but most of the D line trains end at Park Street instead of going ahead to Government Center, Haymarket, North Station, Science Park and eventually Lechemere. Infact many times MBTA serves a separate train between Haymarket and Lechemere. It’s like instead of having one route from stop 1 to 10, we have two routes on the same line from stop 1 to 6 and then from stop 8 to 10.

Next I would like to emphasise the difference between station and a stop. Take for example State Street station. Two lines – Orange and Blue Line have a stop in this station. Blue Line and Orange Line both end up having two stops each in station – one in each direction. Now let’s imagine theoretically, if Eastbound Blue line stops twice in the same station in two different parts of the station (if the station was huge), then Blue Line would have two stops in Eastbound direction with one stop in Westbound direction. That would make Blue Line have three stops in a station.
Not all the stops need to have a station associated with them. A bus stop at the corner of the street is not within a station. We can have more than one route crossing a stop is the most fundamental element. Stop can be associated with route and stop can be associated with a station. Thinking practically, a route cannot exist without stops and a station cannot exist without stops.

Zeitgeist Explorer release

When I am working on integrating zeitgeist with any other tool as a plugin, addon or writing a completely new application, a lot of times I felt the need to run a live monitor of all the events which are being inserted. Since zeitgeist uses dbus, so the obvious answer would be to use dbus-monitor, but this tool is a bit difficult to use in my opinion. When you are working on something which needs zeitgeist, last thing you need to do is to spend next few hours dealing with dbus-monitor.

I ended up writing a small GUI application which connects directly to zeitgeist daemon and receives a notification everytime an event is inserted.  I released it 2 months before, but never got enough time to further work on it even announce it’s availability. The application is very easy and just lists down all the events. Select any event and it will give you all the information about it. Just to clarify, this is a developer tool.

Here is a screenshot.

Zeitgeist Explorer UI
Zeitgeist Explorer UI

The latest release is 0.1 and is available at Launchpad. Please file bugs in case you encounter any. If you a packager, please consider making it available for the distribution of your choice.

Software Programming/Engineering: General Dos and Don’ts

When I first started programming, working on projects both in Open Source and Corporate world – I wish someone made me aware of the Do and Don’t. For a long time, I wanted to sum them up and keep it as a checklist.

Commit Messages

Make sure you put a commit message which makes sense. It is true that a good commit message might be difficult to come up with, but please avoid messages like “Fixed bug” or “Rendering”. Which bug did you fix? What is the bug number? What is the bug about? What rendering? What did you do with rendering? Did you fix it? Did you add rendering capability? Did you remove rendering capability?

It is a good practice to add a line of commit message title, followed by a single blank line and then an optional detailed description of the change which might not be obvious. If you made a commit which alters the working or behavior of the component, you might want to explain why you took that decision in the description section of the commit message

Check In Early, Check In Often

Jeff Atwood has an excellent post on this recommendation. My personal motto is to check-in when I am done for the day and pull in the changes when I start working for the day. I have a friend whose motto was “Every morning, you wake up and do ‘svn up’, even if you haven’t brushed your teeth”.

Keep the change-set manageable

Avoid any such scenario where your commit diff is 2000 lines unless you are nuking a component or any special scenarios. If the component you are working on enters the source tree in a single commit with 4000 lines of diff after 5 days, then STOP. THINK. Is this really the correct way? Can you do it in a better way?

If your single diff contains truck load of different features with 25 files affected, then it is a exercise in frustration for anyone who is attempting to review it. Try breaking it down such that you are able to follow “Check In Early, Check In Often”.

Learning to resolve merge conflicts

To be honest, after working on git for long time, I am still have a sigh face when I encounter a merge conflict. It isn’t always easy to fix it, but don’t forget that merge conflicts do happen and you will need to deal with it even if you collaborate on a piece of software in a very smart manner.

Get your code reviewed

No one’s code is perfect. At times, we are insecure of our coding abilities and become overly protective of our code quality. It took me time to learn that code reviews are not just about improving quality of the code, but it is a great opportunity to learn. Even if you don’t have a better programmer around, get it reviewed by your peers whom you consider at par with you.

Try reviewing code, that helps too – you learn better ways of implementing features and ideas.

If someone criticizes your code, resist from taking it personally. Usually the person criticizing your code has got nothing against you.

Start Questioning

You don’t need to catch your team lead by the neck and demand the answer for choosing this method. Question yourself! If the lead of a project has chosen to use MVC pattern or decided to use Smarty as the templating engine, then don’t just take it for granted. Think over it. Try again! Why does this problem needs a MVC solution? Why does this needs to be implemented in Rails?

A spark of inquisitiveness goes a long way.

Call for Help

Update: $663.05 total. Target reached. Thanks everyone for your help.

Original Post:

Recently I resigned from my job and went back to graduate school for my masters in US. I had to leave my desktop back in India and take my 6 years old along. Even though it works, it is a living horror. I am somehow able to manage my coursework using this laptop and cannot use it anymore for any open source development.

This laptop overheats and burns my legs, has a broken wifi card which forces me to use an external WiFi adapter. The touchpad buttons are broken and using the touchpad requires secial talent.

One of the USB port, optical drive doesn’t work at all. Battery is completely dead. If I pull out the power cord, it switches off the same way as your TV turns off – not even a single second of backup. I recently got a new charger else this laptop would turn into a paperweight. The only things which works are keypad, display and the bare necessary things like RAM, processor to use the laptop.


I know it sounds shameless, but being on a student education loan and in future I might have to use all the sanctioned amount, I am not in a position to buy a new laptop. I have $400 with me and I am aiming for a customized Thinkpad T430 which should last for another 4 years. It might cost upto $1000 including taxes and shipping. If you are happy with my Zeitgeist and Activity Log Manager work, I would like to humbly request you to donate any amount if you are willing. If I don’t make to $1000, I would have to try to figure out other ways to fill the gap.

I have not been able to work on any Ubuntu, Zeitgeist or any other open source project for last 4 months. Partial list of reasons are immigration stuff, figuring out academic requirements and preparations for this change.

If you are in India, you can deposit it in my State Bank of India account. Account number 00000020085201096 and IFSC code is SBIN0004456

What to expect

If I reach the target and get that laptop, these are the things I have in mind. These are the TODO and possible work I would like to do

  1. I would implement the redesign of Activity Log Manager proposed by Matthew Paul Thomas. I was unable to do it for 12.10 cycle due to lack of time.
  2. I would finish the Zeitgeist debugging tool name Zeitgeist Explorer using which you can monitor event insertion in real time and also fetch events based on filters. Lots of more features can be added.
  3. I would finish the Zeitgeist Manual. Most of the work has been done, but there are still empty sections. I would also need to update it for the latest API and changes.
  4. I would work on integration of Zeitgeist with Elementary. Scratch has Zeitgeist logging enabled and I am planning to port gedit dashboard to scratch.
  5. Squashing of bugs in user facing components of Ubuntu like Software Sources, Apport etc.

Privacy and Activity Manager (Zeitgeist) Release

We all love privacy and we take it seriously. Most of the people whom I have met are pretty much vigilant about their data and which makes complete sense to have a tool which can you can use to fine tune your privacy settings. Till now most of our experience with Privacy Settings are related to Facebook which makes me feel like

Layman control center?

If you are a Zeitgeist user, it is pretty expected to expect a Privacy application. Yes, we have it. Today we are releasing it. The first release was a hastily assembled up together application using pygtk (gtk2) which served it’s purpose well for the time being.

Older version of Activity Log Manager
Older version of Activity Log Manager

Soon it was decided in Ubuntu Developer Summit in Orlando that Activity Log Manager will be included in Ubuntu Precise to provide greater privacy controls to user. We improved the looks based on the design provided by the Canonical Design Team.

The interface was kept as simple as possible. This first release lacks a few things as of now and some more UI polish is pending including localization support. Both of them will land sooner.

It has been integrated in Gnome Control Panel (which works only on Ubuntu). In distributions where it cannot integrate in Control Panel, it runs as a standalone application.

Here is how it looks

Entry in GNOME Control Center
Entry in GNOME Control Center
Blacklisting Applications
Blacklisting Applications
Files and Folder blacklisting
Files and Folder blacklisting
Activity Log Manager History tab with Icon
Activity Log Manager History tab with the Icon

The Zeitgeist Team worked on it and ported the application to gtk3 and vala in a few weeks which was a pleasure (and I finally learnt vala. Finally!)

Geeky Juice:

The release is named “Friendly Dolphin” 0.9.0 and can be downloaded from here.

This application would be made available in Ubuntu Precise soon.

If you are building it yourself and want Control Center integration, then pass –with-ccpanel to the configure script.

./configure --with-ccpanel

This would work on all the Linux distributions which derive from Ubuntu and don’t change the gnome-control-center packaging since this integration relies on a Ubuntu specific patch for control center.

Please log any bugs which you come across. To make sure that we are able to help you, please provide detailed explanation of what led to the issue. You can file your bugs here

Ubuntu Power Users reorganization – An advice to our new roletaker

Jono founded Ubuntu Power Users group a few months back. Since then lot many things were discussed but due to being a bit late in the cycle.

Power Users Group got a fresh new start and it was advised to hand over the bigger responsibilities to James Gifford. If any of you don’t know him, he a a very young yet highly energetic community member and an active member on AskUbuntu.

James, every initiative has it’s high and low in activity. When Power Users was started there was many good discussions but now it looks like the team really needs a strong leadership. Do not be demoralized if you find that you are the only person who is working under power-users banner. There might be other people interested in contributing, but they might not feel like contributing when they find themselves alone. It is always more encouraging and consoling to find that the project you are contributing to has an active leader and your work is actually going to make a significant impact. When there is a leader who is working tirelessly, I feel more confident because even if I am not able to contribute, still there is someone who is making sure that my work will reach the wider audience.

Being a leader is tough. Being a maintainer of a software project is also very tough. It is sometimes easier to be a person who just commits code than being a maintainer. You have to make sure that all the nuts and bolts are assembled together in the correct order. Contributors might only make separate pieces, but you have the responsibly to assemble them together and make a working product. Sometimes you even have to make sure that your assembled piece works properly when it itself has to act like a nut to an even bigger piece of software.

I admit having the feeling of “Why? Why am I the only person who is working on it? Does it mean no one else is interested?”. The answer is that probably many other people might be interested, but they are just too busy in another piece of software or community which they maintain. They might be not getting enough time to contribute to the project in which I or you work.

That said, James please make sure that atleast Ubuntu Tweak is included in Ubuntu archives starting Precise. The Ubuntu developers had their own valid share of criticisms for this product, hence it is time to address the issues. I can help you in support and documentation part (on AskUbuntu).

Best of Luck

Zeitgeist User Survey 2011 Fall Results

Hello everyone,

As I announced earlier we had opened User Survey for Zeitgeist for general feedback and assessment of our own work. It was also to help us understand what are the user requirements.

After 5 days, we have received 261 responses. Instead of going in details and boring you, I would like to keep the post as short as possible and provide detailed analysis in the next post.

Our survey contained questions related to general Zeitgeist questions, Activity Journal, Activity Log Manager, Datasources and Community aspects. Users were also asked about their distro of choice and optional name, age and Country.

Distribution demographics

The biggest share of responders were using Ubuntu – 136. 30 of them were using derivatives. I am also happy to see Fedora and Gentoo in the list.

The other 10 distros included Mandriva, Chakra, Kubuntu (which could be part of Ubuntu and it’s derivatives)

How did you come to know about Zeitgeist

Online blog/news-outlet and Planet make up a huge percent of users. It is evident that people do follow these two very closely. Notice that total does not add up to 100% as just hearing the name might not be sufficient for someone to figure out what it is.

How much do you know about Zeitgeist

Pretty much obvious conclusion. The user know what it is but most of them are not geeky. Looks like those 6 “I hacked on the codebase” was selected by contributors.

Do you know what is Zeitgeist Datasources

One half of my brain says “Teach them what it is” and the other side says “No. Not everyone is very interested in being force taught”. Guess what? I take the central stand and will publish a short article on what all components of Zeitgeist do.

Which of these Datasources do you use?

This is a VERY VERY important chart. This tells us the adoption of datasources esp when datasources are not present in Ubuntu, Debian, Fedora and openSUSE (AFAIK for last two). I think they are present in Gentoo and Arch.

I am really humbled that so many people like Firefox datasource even though it had suffered so much with regular xulrunner API breakage. With our new datasource which uses js-ctypes, all the API headache is gone. Gedit. Banshee and Chrome too score pretty high

Which datasources would you like to see in the near future?

I never thought Thunderbird would come out as first. My expectations were with Gnome-0shell which is actually second.

Good news is that Thunderbird datasource is in progress. Mark Jtully is working on it. Gnome-shell would is also in progress.

Since other were 76, I would like to name a few, though I would prefer a separate post for this. The number 76 has led me to make up my mind for a special treatment.

Let me list a few:


Jason. Is that you? If yes, then contact me on IRC 🙂


Personally my favourite too


Work in progress


Telepathy plugin already present


Even I am excited about it


Work in progress. Stay tuned for the next blog post

Heard about Activity Log Manager

Activity Log Manager is Administrative tool for Zeitgeist. You can temporarily stop logging, blacklisting certain applications from logging information, blacklisting certain mimetypes, clearing all history between time range etc. It is a handy tool. A new version is going to be out soon.

It is really sad that half of the people never heard about it. Looks like we failed at spreading the word. By the way there has only been a single release.

How much are you involved with the community?

40% of the people are users. “Nor am I interested” is not a harsh phrase. It simply means “I use so many applications and it is not possible for me to be involved in each and every community”.

15% of the respondents are eager to join the community. Probably you all can help us in testing. This is where most of the manpower is needed.

In case you have problems, where do you ask your questions?

Two-third of the people do prefer forums. Mailing lists and IRC still trump over Q&A sites. I think many people have answered this not only for problems but also for general questions.

In case you are an Ubuntu user, you can ask your question on Ask Ubuntu and tag the question with zeitgeist tag.

Are your problems related to Zeitgeist solved?

Did I tell you that this chart has boosted my morale? 70% of the people didn’t have problems with Zeitgeist. I now really guess that these people were asking general questions or discussing about zeitgeist and replied to the above question.

Whatever be the case, we still have 10% unsatisfied users from the samplespace. 10% is a significant number. Hope we can bring it down by the next survey.

Few interesting comments we received

More focus on the related items of Zeitgeist

Looks like we really lacking here. Thanks for reminding

The ability to put zeitgeist in private-mode

It is already present in Activity Log Manager

I’m a software developer and do lots of testing every day. I need a way of blocking activity from my development/testing trees

There is a “Logging Active” toggle button in Activity Log Manager

 It[Activity Journal] needs to be abstracted into Unity

Isn’t it already? Probably the implementation is different. There is no journal view in Unity

It doesnt show all my files

You need more datasources. In other words we need to provide more datasources which you need to install.

It’s [Activity Journal] good, but shouldn’t we be providing this information through the UIs made available by the host OS?

It is somewhat done for Unity. Next up in Shell and Plasma. In any case none of them show the Journal view as Activity Journal

I really like being able to see what I did, and when I did it. The thumbnails for code snippets, videos, and pictures are also rather nice. Congrats on making a easy-to-use computer journal 🙂

All I can do is smile

Please integrate it into Gnome Shell if it light enough.

This is also my personal wish. It might take some time

It would be nice if these datsource plugins are included by default without the user having to do it manually. This way it just works.

This has to be done at distro level. This is something even I would love to see

A ppa would be nice.

A you wish, but remember there can be dragons

Releasing Zeitgeist Datasources 0.8.0

One behalf of Zeitgeist Team I am happy to release zeitgeist-datasources 0.8.0

Zeitgeist Datasources are a collection of extensions/ addons/plugins which log the activities of the user and insert it in the zeitgeist engine which can be later used for lot of useful tasks. These extensions insert lots of extra events apart from the ones logged by zeitgeist-datahub

The extensions present are

  • Eye of GNOME
  • gedit
  • Totem
  • Rhythmbox
  • Tomboy
  • Chrome
  • Emacs
  • Firefox 4.0
  • Geany
  • Telepathy
  • Vim
  • XChat
  • Bazaar

Get the release tarball from here

Few of the extensions require Zeitgeist 0.8.0 to build.

Launchpad project Page

More info about Zeitgeist

Package for Ubuntu would be available on Zeitgeist PPA. I have built it in my PPA, it needs to be tested and then copied to the Zeitgeist PPA. If you have added the PPA, then you should get the update


GNOME.Asia was held in Bangalore from 28th March to 3st April. It included a hackfest from 28th March to 1st April and two days of GNOME.Asia summit on April 2nd and April 3rd.

On 1st of April I attended AndroidCamp and was drenched by the time I came back home. only to find myself not well to attend on the first day of the summit. The first day was cut short to half because you Cricket World Cup Finals. India was in finals, the summit is in India and India is a country where cricket is religion and you expect people to turn up? 🙂

As I heard from Manu, the first day turnout was huge. That day India won the World Cup (YAY!). Next day I somehow managed to get up and head towards the venue (only 30 kms from my home).  I was one of the few people who reached the venue before 10AM on Sunday.

The first session I attended was from Allan Day on Designing the next generation desktop. Then I proceeded to another hall where Andre Klapper was giving his talk on Contributing to GNOME Documentation. He had some examples which was more easy to understand than plain talk on concepts and theory.

By the time Andre finished his talk, I met more friends – Yuvraj, Sindhu and Shashi. Next talk was from Yuvi who was talking on GStreamer 101. He explained GStreamer from a newbie perspective and slowly his talk started carrying more geeky explanation (expected). He continued even after his session got over by explaining his he got into GSoC on the Cheese project. He is the REAL motivator 🙂

When Yuvi finished his talk, Manu and Sankar were waiting for me outside the hall. This is the first time I met both of them even though Manu is from my college. There were more students from my college – two names I can remember is Vikas and Saurabh. Looks like LUG Manipal bore some substantial results.

After these session, I had a golden moment of my life when I went and met Vincent Untz. He is the current Godfather of GNOME. We both talked a bit on Zeitgeist when Allan Day and Andreas Nilsson also joined the conversation for a minute or two before we all moved for lunch.

The only two talks I attended in second half was on GDK(entered late so didn’t remember much) and another on DLNA in GNOME3 world by Arun Raghvan. I could not attend any other talk because I came out to talk to our friends when we met after a long time.

I am a bit sad that I missed a few talks. One of them was by Manu on Libyui and Contributions made by a newbie to free desktop’s empathy by Chandani Verma. I should have stayed back and also attended the Lightening talks. All the slides are available here

The event was well organized. I could not see any kind of chaos or anarchy. The openSUSE stall was filled up with students asking for it’s CD. I saw many students trying out openSUSE and guess what – most of them had Ubuntu installed. Whenever I glanced here or there, I could see people using Ubuntu everywhere 🙂

Android Camp Bangalore

I got up in the morning only to find there was no electricity. Not sure if this is somewhat related to April Fool prank, but was enough to piss me off.  Checked my watch – 9:30AM and the unconference is supposed to start in half an hour! I needed to get ready and travel more than 18 kms. The next one and an half hour involved running all around the house to get ready, gulping down breakfast, hunting for buses and then finally messing up with an autorickshaw driver when he started using his random number generator for determining the fare.

Finally I managed to reach the place by 11AM. The event AndroidCamp was of an un-conference style. Upon entering the first bad news I got was that I missed a session on Arduino which went excellent.

The rest of the first half session were pretty boring except one from Kashif Razzaqui. His talk was on his experience when developing apps using cross-platforms development tools like Titamium and PhoneGap. He nearly blasted off on the technologies which I think all the geeks love to do. I can see from everyone’s face that they were really enjoying his session. He pointed out how developing anything with Titanium is a total PITA. He did use some funny and amusing quotes like

Those Apple fanboys have to use objective C which cannot be even called a programming language

If Java and XML gives you a hardon, then something is really wrong.

Not sure if both of the quotes is exact as he said, but the audience did chuckle and smile at these jokes. No one at the event did a “Well Actually”

Then Kiran told us that the food van had got lost and the driver is circling the neighborhood searching for the venue. How not, even we all had so much of problems locating the venue. During that time Kashif put up a video of Android during Google IO.

After the lunch when all of us are supposed to sleep, two sessions caught everyone’s excitement. Since I was in the “Geek” room I missed the Firmware and Rooting talk in “Suits” room. Later two sessions happened in Suits room – “Android-scripting” and “Android Rants”.

The session android-scripting was taken by Sajjad ‘geohacker’ Anwar. He is the person who works on OpenStreetMap, writes python code and contributes to Ubuntu Accessibility. He explained how and when scripting can be useful using python than writing Java code. People put up points about speed, performance of this idea, all of which was answered by him. Good job.

Sajjad Anwar presenting android-scripting

The next session was about “Android Rants, The reluctant skeptics” by Supreeth which was basically a session to tell what all issues you have with Android. Some of the topics discussed were

  • Battery
  • Upgrades
  • Fragmentation
  • iOS and Apple as a competitor
  • WP7 as a competitor

I think this was the topic where people took part with utmost interest. Comments came from all the sides of the room. Supreeth had to stop them many times to make sure that the discussion did not turn unruly or offt-topic. I don’t think any aspect of mobile ecosystem was left out during this talk which includes Android, Google, Motorola, iPhone, iPad, Blackberry, Nook, Windows Phone 7, Nokia, Palm, HP and every other entity even remotely associated with mobile phones.

The last session was the Feedback session. This was the session where everyone ended up speaking even though I thought that only a few people might express their opinion. The thing most requested was keeping such events on weekends as getting day-off is really tough for most of the people especially when it is Friday.

All the day I kept a track of the April Fool pranks starting from cyanogen quitting Android to GNOME3 getting delayed to Google Motion. On returning home, I stumbled across one excellent-brilliant-kickass article I am Jef Spaleta by Jono Bacon. This was the day! Lastly if you didn’t notice even WordPress has a small prank (for wordpress.com hosted blogs)