I usually don’t post articles about current affairs. However, a recent series of events has inspired me to write about this.
Towards the end of January, the president of our company – iScan Online, Inc., was complaining that our service was experiencing major slowdowns. I investigated the issue, but I couldn’t find anything wrong with our production environment. We were stumped.
One evening I also noticed a slowdown while using our service from my house. I realized that the one thing in common between me and our president was that we both had FiOS internet service from Verizon.
Since we host all of our infrastructure on Amazon’s AWS – I decided to do a little test – I grabbed a URL from AWS S3 and loaded it.
I also noticed that our Netflix streaming quality is awful compared to just a few weeks ago.
Next, I remoted into our office – about a mile away from my house. I tested the same link –
So I contacted Verizon support over their live chat.
Verizon had me do a speedtest.
He says “You have excellent Bandwidth – is there anything else I can help you with?”
I replied – “Yes. Why are these files slow…”
So he proceeded to walk me through various troubleshooting:
“reboot your router…”
“make sure your system has latest updates…
“change your wifi channel”
After about 30 minutes of this – I grew impatient. I explained to him that there was something limiting the speed on their side. He remoted into my system with a screen sharing tool, and I showed him my remote screen to the connection at the office. He kept on saying that bandwidth is different for different locations etc…
That’s when I decided to press him. Here is a screen capture of the final part of our chat:
Frankly, I was surprised he admitted to this. I’ve since tested this almost every day for the last couple of weeks. During the day – the bandwidth is normal to AWS. However, after 4pm or so – things get slow.
In my personal opinion, this is Verizon waging war against Netflix. Unfortunately, a lot of infrastructure is hosted on AWS. That means a lot of services are going to be impacted by this.
PS> a number of folks have questioned the expertise of the support individual. I completely understand. I’m not a networking expert, but I did want to share 2 more pieces of data that I think are significant:
Traceroute from Residential Side:
Tracing route to iscanonline.com [188.8.131.52]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 8 ms L100.DLLSTX-VFTTP-65.verizon-gni.net [184.108.40.206]
3 10 ms 6 ms 9 ms G0-5-2-0.DLLSTX-LCR-21.verizon-gni.net [220.127.116.11]
4 16 ms 9 ms 10 ms so-5-0-0-0.DFW9-BB-RTR1.verizon-gni.net [18.104.22.168]
5 10 ms 9 ms 9 ms 0.xe-3-3-0.BR2.DFW13.ALTER.NET [22.214.171.124]
6 9 ms 10 ms 9 ms 126.96.36.199
7 10 ms 9 ms 10 ms ae-1.r08.dllstx09.us.bb.gin.ntt.net [188.8.131.52]
Traceroute from Business line (1 mile away)
traceroute to iscanonline.com (184.108.40.206), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 18.036 ms 1.326 ms 2.318 ms
2 l100.dllstx-vfttp-93.verizon-gni.net (220.127.116.11) 5.870 ms 5.211 ms 5.193 ms
3 g0-5-0-2.dllstx-lcr-21.verizon-gni.net (18.104.22.168) 7.400 ms 67.679 ms 10.605 ms
4 so-5-0-0-0.dfw9-bb-rtr1.verizon-gni.net (22.214.171.124) 12.062 ms 6.652 ms 17.799 ms
5 0.xe-3-3-0.br2.dfw13.alter.net (126.96.36.199) 7.207 ms 7.858 ms 9.616 ms
6 188.8.131.52 (184.108.40.206) 7.435 ms 7.256 ms 10.366 ms
7 ae-1.r08.dllstx09.us.bb.gin.ntt.net (220.127.116.11) 7.365 ms 10.160 ms 9.083 ms
So I got a Raspberry Pi for
Christmas (thanks Secret Santa!), and I decided that I wanted to
create a little retro gaming appliance with it.
First thing I did was install the NOOBS
software that is recommended for first time Raspberry Pi users.
Next I googled for gaming emulators on Raspberry Pi. After grokking
several forum and blog posts, I came across the RetroPie project
(http://blog.petrockblock.com/retropie/). This certainly looked
After going through the RetroPie-Setup scripts – I thought that I was
almost home. So I started up emulationstation – a graphical frontend
for all of the emulators. Once I did the obligatory startup of Super
Mario Bros – I was sorta happy. However, I thought – keyboard
controls suck. I want a game controller.
Being kinda cheap, I said to myself – “hey…my son has a bunch of
game stuff up stairs…” – after digging though his gear, I found a
nice GameCube controller.
I Googled for “GameCube controller
raspberry pi” – which led me to
Installing the gamecon_gpio_pri
I dug in a bit deeper and looked at how I would wire up this
controller. This site
describes the interface for the Revision 1 Raspberry Pi board.
However, I needed a wiring diagram for the Gamecube controller so that
I could hook it up.
I was super excited at this point and figured I was 90% of the way there!
I wired up Pin 2 of the GC controller to GPIO 2 on the Raspberry Pi.
I also tied Pins 3, 4 and 7 together and wired them to Ground on the
Raspberry Pi. Finally, I hooked up the controller’s Pin 6 to VCC+
which is Pin 1 on the Pi. I used this site for
So once I had the GameCube controller wired up, it was time to load the driver. After some experimentation, I figured out the command to load the driver for GPIO2:
sudo modprobe gamecon_gpio_rpi map=0,0,0,0,3,0
The modprobe command is a Linux command to load and unload kernel modules. The map parameter is described in the source code:
I clicked a couple of buttons /moved around the joysticks and saw that the values were changing. So I figured I was done with the controller setup.
After removing ~/.emulationstation/es_input.cfg (this causes emulationstation to run the input setup again) – I noticed some strange things:
When the program asked me to press “Up” – It would jump 2 or 3 steps through the process after I pressed up.
After 2-3 times of running the setup (carefully timing my “Ups” and “Downs”) – I tried playing a game
The game was not responsive and didn’t seem like the controller was working correctly.
So I went back to google and started doing more research.
Unfortunately – there wasn’t much information available. Nobody
seemed to be having my problem, but there also wasn’t much discussion
anywhere regarding the GameCube controller and the Raspberry Pi. I
posted a question on the raspberry pi forums about this just to see if
I would get a response. But being impatient…I wasn’t going to wait
for the answer.
Hacking on the code
After digging around the file system on the Raspberry Pi a bit, I was able to find the source code for the gamecon_gpio_rpi kernel extension. It doesn’t appear to live in github or any other source code repository. Although there is a deb package that includes the source. If you’ve installed the driver with the RetroPie-Setup script, the source lives here:
It is managed with dkms – so hacking on it is relatively straightforward. So I started digging in a bit to the source.
After looking through the source a bit, the first thing that seemed incorrect was this:
#define GC_GCUBE_REQUEST 0x40c002U /* the request data command */
The reason this seemed incorrect, is that there were a few articles written by other folks regarding the gamecube controller protocol. The consensus was that the command should be:
#define GC_GCUBE_REQUEST 0x400302U /* the request data command */
So I figured I’d start there and see where that got me. I copied the whole src as follows:
And I tried my change to the GC_GCUBE_REQUEST hoping that it would solve all my problems.
After rebuilding and loading the module I ran the jstest program again. It seemed to be working a bit better, but it still wasn’t as responsive as one would expect a controller to be.
I was honestly a bit stumped at this point. I double checked all my connections and couldn’t find any problems.
So I went ahead and brought out my trusty oscilloscope. Based on the description of the gamecube controller protocol from other articles – I expected to see a 24 bit message (23 + stop bit) containing the following:
1000 0000 0000 0110 0000 010 1
So here was the capture from my scope.
After annotating it with 0s and 1s:
So after looking at that – it looked pretty close. And given the fact that I was receiving controller information, I figured it was probably correct. However, I figured I needed to do a little more analysis. First, I measured the timing of the 0s and 1s of the response from the controller. I captured a successful 64 bit message returned from the controller and measured the time of single bit:
It measured at 4us – which matched what other articles I found while doing research. If the response was 4us per bit – then the command sent from the gamecon_gpio_rpi driver should match. So I measured the command signal from the Pi:
Doh! It measured at 2us. That doesn’t seem right…so I dug into the code that sends the command over the gpio pins:
We can see here that the code is correct if udelay is calibrated at 1us (1 microsecond). This code appears to be correct because there is a 3us pull low followed by a 1us pull high for a 0 bit. And there is a 1us pull low followed by a 3us pull high for a 1 bit. However, the scope indicated that the delays weren’t matching.
So I decided I would fiddle with those values a bit to see if I could get the command to match up with a 4us duration per bit. After some trial and error here was the code I arrived to:
At a glance this didn’t make a lot of sense because the values aren’t clean multiples. I don’t have a good explanation for this at the moment.
But the capture from the scope showed that I had achieved a 4us duration for the command duration per bit:
So I ran the jstest – and all seemed to be working properly – the updates to the joystick and button data seemed to be good. However, there was something strange happening on the scope.
It seemed that the response was intermittent. It would rapidly jump between these 2 states:
I felt that it should not be jumping back and forth between having a response sometimes and not having a response other times. So I took another look at the code that sends the response and noticed this:
/* send stop bit (let pull-up handle the last 2us) \*/
So based on my previous change to the other timing, I changed that code to this:
/* send stop bit (let pull-up handle the last 2us) \*/
After rebuilding the module one more time, it appeared to stabilize the signal so that I saw a nice constant wave form that wasn’t intermittently receiving responses – but rather receiving responses every time a command is sent.
I suspect that the timing of the udelay has changed with various changes to the raspberry pi and/or Linux kernel. It appears that udelay supports resolution of about 500ns (.5us) based on my experimentation. However, this might change from version to version – or worse – it might change based on the load of the system. So I will have to do further testing with actual gameplay to see if this is going to work long term or not. Unfortunately – the raspberry pi running Linux is NOT a realtime OS.
In a previous post – I talked about using an STM32F103 based Cortex M3 with OS X. Since that post, I’ve ordered one of these – STM32F4-Discovery. I am thinking about porting the work done with smoothieware from the LPC17xx platform to the STM32xxxx platform. The STM32F4-Discovery is a relatively new development board that shows a lot of promise. At ~$15 it is a bargain platform with a lot of features.
As usual, I need all of my goodies to work on my Mac and Windows boxes since I go back and forth a lot. The following is a quick guide to getting this up and running with OS X Lion.
I wanted to work on some ARM Cortex M3 programming on my Mac…
In order to program my dev boards, I use a Segger J-Flash. At first glance, they don’t appear to support the hardware on OS X. However, after some digging I found that they offer a beta that includes OS X support here
Once I got that installed, I received the following error:
dyld: Library not loaded: /usr/local/lib/libusb-1.0.0.dylib
I have deliberated for a while now on what to do with my blog. I have not been particularly active in my blogging, but I have definitely been thinking about a number of things I need to blog about. However, I found that my blogging engine (Typo) was very limiting for me. It turns out, that most blogging engines are limiting to me. I am not very comfortable using a web UI for my workflow! I am a coder that uses SVN, git etc…day in and day out for my daily work. So I asked my friend, Glenn Vanderburg, if he had any ideas – and he suggested Octopress.
So far, it seems that Octopress really fits the bill. I generally like using a text editor and version control for everything – including documentation. I am somewhat of a fan of wikis, but I prefer having raw access to my source trees. So Octopress really allows you to hack away at your blog, and then push and deploy just like you would any other software.
Migration was easy since I didn’t really have that much content to begin with. I simply cut and pasted my blog content into new articles generated through rake. I’ll recap the steps I took:
First, I forked Octopress from github
Next, I edited _config.yml to reflect my site configuration…this is pretty self explanatory.
Then, I generated new posts via rake new_post["..."]
I edited those posts to reflect the original publication date from my former blog
Finally, I tested everything using rake preview
Once everything was to my liking, I edited my remote server nginx configuration to point at a directory I created to hold my content.
Restarted nginx on my remote server
ran rake deploy
BAM! It was done.
I absolutely love this new approach to my blog so far. It is static content!!! No database…at least not for now. I hope that this can stay as static as possible. I really just want a place to call my home page where I can rant and write about my hobbies and code etc… without thinking about another web application that I have to manage. Not to mention – I don’t like working through a web based workflow to being with, so this really lends itself to a hacker style blog! I’ll just have to get up to speed with all of the various formatting options…