How many people are capturing BCU data?

What I'm working on is actually a web-based tool, where any CMC owner could upload their data, look at other users data, see interesting statistics, etc. I can do all that with my own data, but it's more interesting the more data there is.

What I have so far:

• MySQL database structure
• Scripts to load a BCU log file into the database, then aggregate the data into 5s samples
• PHP/HTML code to show the most interesting rides, and links to display those graphs

By 'interesting', I mean things like

• Highest speed attained
• Highest current draw
• Highest regen current
• Longest ride
• Lowest State Of Charge
• Highest throttle position

I've got all those things worked out already, including the graph that I showed a sample of above. What I still need to work on is

• File upload from the browser
• Show a table of interesting stats per user (with links to the graphs)
• Allow the viewer to generate a graph for a particular, arbitrary BCU log dataset of their choosing
• Allow the viewer to choose what data to include in the graph, i.e. enable/disable graphing throttle position
• Registered user management

I also need to determine how best to make this available if there's interest. Right now I'm doing my development on an old netbook with a Celeron processor, 512MB or RAM and 2GB of disk; the load scripts take a few minutes to import a log for a 30 minute ride, but displaying the web page or graphs is sub-second so the hardware requirements are minimal. I would also need to make it accessible to other folks; I'd rather not open up my home connection for direct internet access, but I might consider that for a proof of concept. I would also be willing to pay to put it up through an ISP with its own domain (cmcowners.org perhaps?), but again only if there's interest.

I'm open to comments & suggestions.

And I agree that you can generate graphs with Excel, but I ran into the problems MEroller discussed; with samples taken every quarter second, there's a lot of datapoints in a BCU log, and it's hard to get Excel to chart anything interesting (at least for me).

LeThala,

A lot of the fields I don't know what they represent, and I suspect that they're irrelevant to me for the most part; things like kick stand status, charge door status, left and right brake status, etc. I haven't bothered to reverse engineer those, but here's the field definitions I believe I do know (any field labled f# or F# is a placeholder for a field I don't know):

"f0","Time", "State","f3","f4","f5","f6","f7","f8","f9","f10","F11","F12","pack_v","cell_v","throttle",
"temp_1","temp_2","temp_3","temp_4","current","F21","speed","trip","F24","SOC",
"F26","F27","F28","F29","F30","F31","mileage"

It might make more sense to look at it from the database definitions (I'm just cutting and pasting from my SQL script):

F0 char(4) NOT NULL DEFAULT "",
bcu_time time NOT NULL DEFAULT "00:00:00",
state char(20) NOT NULL DEFAULT "",
F3 char(4) NOT NULL DEFAULT "",
F4 char(4) NOT NULL DEFAULT "",
F5 char(4) NOT NULL DEFAULT "",
F6 char(4) NOT NULL DEFAULT "",
F7 char(4) NOT NULL DEFAULT "",
F8 char(4) NOT NULL DEFAULT "",
F9 char(4) NOT NULL DEFAULT "",
F10 char(4) NOT NULL DEFAULT "",
F11 char(20) NOT NULL DEFAULT "",
F12 char(4) NOT NULL DEFAULT "",
pack_v decimal(5,2)NULL DEFAULT "0.00",
cell_v decimal(4,2) NOT NULL DEFAULT "0.00",
throttle decimal(5,2) NOT NULL DEFAULT "0.00",
temp_1 decimal(5,2) NOT NULL DEFAULT "0.00",
temp_2 decimal(5,2) NOT NULL DEFAULT "0.00",
temp_3 decimal(5,2) NOT NULL DEFAULT "0.00",
temp_4 decimal(5,2) NOT NULL DEFAULT "0.00",
current_draw decimal(5,2) NOT NULL DEFAULT "0.0",
F21 char(4) NOT NULL DEFAULT "",
speed decimal(4,2) NOT NULL DEFAULT "0.00",
trip decimal(5,2) NOT NULL DEFAULT "0.00",
F24 char(4) NOT NULL DEFAULT "",
SOC decimal(5,2) NOT NULL DEFAULT "0.00",
F26 char(4) NOT NULL DEFAULT "",
F27 char(4) NOT NULL DEFAULT "",
F28 char(4) NOT NULL DEFAULT "",
F29 char(4) NOT NULL DEFAULT "",
F30 char(4) NOT NULL DEFAULT "",
F31 char(4) NOT NULL DEFAULT "",
mileage decimal(8,2) NOT NULL DEFAULT "0.00"

I haven't figured out which temperatures go with which sensors, but I think I have an idea in general what areas the sensors are for (battery, power lead, motor, controller)

I'd be more than happy to collaborate on this, although I have one rule: you can't laugh at my code (derisive snickers are OK). I'm a tinkerer/hack, not a professional; I do stuff with the LAMP stack as an amusement.

Unless someone else pipes in that they'd like to collaborate, maybe we can start an email chat to discuss logistics and code. Let me know what you think.

MikeB,

Other than the first and last fields, that matches up perfectly. Of the BCU data I've loaded into MySQL (30 logs, 766K rows), the first field is always a "D" and the last field appears to be total mileage. It could be a difference in BCU version; I think I was the last test pilot to receive their scooter, and mine's been back to CMC for the ugpraded charger plus replacing some bad cells in March/April so the BCU may have been updated.

Regardless, this is excellent data to have; thank you! Now that we have an idea of what the temperature readings are for, I might add an option to show those on the graph as well.

OK, I have a workable proof of concept up on a free hosting website:

http://cmcowners.comze.com/bcuplot.php

I only have my data right now, but if anyone wants me to upload their data please send me an email and we can work out the details. If there's interest, I'll work on allowing people to load their own data through a web interface (the upload script is written, but needs to be changed to work from a web page).

The data is 2 second averages of the BCU logs, since the 4 samples per second that the BCU captures is too granular to plot meaningfully. I've tried to put as much critical data in the charts as possible, with an option to add secondary data like pack voltage, temp reading, throttle position, etc.; I don't want the chart to be so busy that it's impossible to read. (Apologies to anyone who's color-blind; it may be hard to discern which line represents what data. I'm open to suggestions on that if it's an issue).

If it's not obvious when looking at the chart, the second graph at the bottom is a representation of various on/off conditions over time: front brake, rear brake, low speed switch, LVC. I only added those items I thought were important when riding, and left things out like 'charge door open'.

I'd like to add a user profile section as well (geography, typical ride, personal stats, favorite ride, etc.), but I thought it more important to start with the actual BCU data. I'm sure there are lots of things that I haven't thought of or included, but this should be a reasonable start.

I had an idea last night and had some time today at lunch, so I added a page of 'widgets' that can be used to embed images generated from the BCU database. If you check my signature (at least as of now), the image embedded there is generated directly from the database on the website. The same page can be used to generate an image of a particular riders top speed. For either one, all you have to do is pass it the rider number and the option you want (SPEED or MILEAGE).

I have a page of samples at http://cmcowners.comze.com/sample.php.

As always, if there's interest, there are probably lots of cools things we could do with the data. I'm just throwing things against the wall to see what will stick, in the hopes that perhaps we can build a better and more interesting user community for ourselves.

PJD,

I agree that looking through the CSV output of the BCU logs is difficult; that's why I thought that a utility to generate a graph would be useful. For instance, in the graph for my longest ride:

I can see that from three minutes until almost eleven minutes, as I was driving down M-140 at just under 50 MPH, I was only drawing about 33A. I find that interesting, as on my last scooter I would have been drawing 70A to hold 35 MPH. Then at twenty-seven minutes into the ride as I got onto I-94, I hit 66.97 MPH, drawing almost 170A with the pack voltage dipping to 90V (the only way for my last scooter to reach 66 MPH would have been in freefall).

It's probably just me, but I like looking at this stuff, and trying to figure out how I can improve my ride efficiency, optimize my charge patterns (I'm a believer in using the middle 60% of a Lithium battery), etc. I also think it would be useful for someone considering purchasing a C130 to know that a 6'7", 280lb guy (305lb with gear & luggage) can drive 21 miles at around 50 MPH and peaking at 66 MPH, and only use up half the range of the battery pack (this ride started at 90% SoC, as it was the second part of my commute that morning).

But I seem to like tilting at windmills; I do it all the time (anyone need a few hundred 'Ron Paul 2008' flyers?). I'll not be offended if the overall reaction is a resounding 'meh'. It's cost me nothing monetarily and I learned some new programming techniques that I'm certain I can use later. But maybe someone will find it useful, or be inspired to build something better/cooler/prettier. I'm just throwing ideas out there so that those of us who have these scooters can better share & compare what we've learned.

Blergh. I should have double checked that before I posted it; 33A for 50 MPH should have set off alarm bells. But still, it tells me I'm running at about 1 C to hold that speed, which means I'm being quite kind to my battery pack.

And interest in this type of data could be an 'engineer' thing as MEroller suggests; I have a paper around here somewhere that says I graduated with an enginering degree, although I certainly wouldn't claim to be one at this point in my career.

John,

Considering you represent 50% of the existing users, I think we can make some adjustments to accommodate you. :) Strictly speaking, I don't need the date at all; the rider and ride numbers are what uniquely identify the data from a particular ride. If you think it's too revealing to have the date, I can remove that easily enough (or at least not show it by default). Would you want some sort of free-form comment area for the ride, a pick list for day of the week designation, or something entirely different?

I'll see what I can work up. I already have a table with one row for every log that I use to record interesting stats, so it shouldn't be too hard to allow someone to add some comments or designations of their choosing for a given log. The tricky bit will be determining the best way to handle user security, authentication & authorization. I typically use LDAP authentication, but I doubt that my free service provider has that. I'll have to write something to take care of that, so perhaps if it's just you and me I'll just use htaccess for now and trust you to upload and edit your own entries, rather than writing a full blown AAA enabled administration webapp right now.

I have rudimentary logfile upload & process pages done. I was out of town for a while, so I haven't had time to add any of the new stuff John asked for, plus I want to add error checking and some more logic to the log processing, but it's there and it works now. If anyone wants access to upload their own log files, send me a email or private message, and I'll set you up with a user ID and password to use the new pages.

Aaron, I'd be happy for the assistance. I have no issues with your students reviewing my code or data (or making improvements?); I'll have to ask LeThala whether he minds having his data reviewed. I sent you a private message yesterday, but I thought I should respond here as well; I'll try sending you an email later so that we can discuss it outside of the forum.

While I'm posting, I might as well address a few updates I made to the site:

• I've created enough structure for rider profiles to allow the page to determine if it should display sensitive information like ride dates, based on rider preference. I don't have the code written yet to allow the user to modify their profile, but I've set it in the DB and changed the existing code to check the DB for the setting.
• I've streamlined the method of choosing the log to display. Instead of rider, date and ride number, it's now just rider and ride number. If the rider allows the page to display private information (like the ride date), the date of the ride will be show with the ride number in the drop-down menu.
• I've created rider #0 for people to play with, if they want to try the site without getting an ID from me. I don't yet have code written to allow them to delete their data once uploaded (that's on its way), but if you want to upload a log as user 0, you can. It is set to NOT display the date. The pages are still crude, but they work. To do so, use the following procedure:

Go to http://cmcowners.comze.com/admin/upload.php

Log in with User ID '0' and password '000Destruct0'

Select a log file from your PC to upload. It must be less than 2MB in size, and have a naming convention of 'LOGxxxxx.TXT' or 'logxxxxx.txt' where xxxxx = the log # (this is the file naming convention of the openlogger; I can modify it if necessary).

After uploading the files, go to http://cmcowners.comze.com/admin/process.php. Choose the file to process, a date for the ride (optional), then click the 'Process' button. It should return after a few seconds, telling you that the file was processed then deleted.

NOTE: Ride numbers are unique per rider; rider 0 can only have one ride #1, one ride #2, etc. Just rename the logfile to something else and reload it if that ride # has already been processed for rider 0 (I realize full well that the possibility of this happening with the miniscule user base is minimal; just thought I'd mention it).

As long as there's interest, I'll keep plugging away at it. I'm still working on the items LeThala requested, but if there's other things people would like to see, let me know.

I've made some more changes to the site. I've rewritten the user and data management, giving the rider the ability to do the following themselves:

• Upload log files
• Process log files
• Manage DB data (delete & update)
• Delete uploaded files
• Change password (not available to test ID '0')
• Update profile (not available to test ID '0')

I've also tried to incorporate some of LeThala's requests by adding the following:

• Riders can add an arbitrary comment to a ride
• Riders can provide profile information about themselves
• Riders can list a 'favorite ride' and enter a special comment about it
• Riders can choose whether private info (ride dates, comments, profile information) is displayed to the world

You can see these things for my rider ID, #34. With all the additional data, the main display page is getting a bit busy; I may need to figure out a way to clean it up or reformat it. I'm debating whether to show a table of every ride, with key data and comments, for a given rider on the main site. It can get rather long (I have 98 rides under my ID), and wasn't sure that it made sense despite it being the best way to see all the ride comments.

In order that it can more easily be moved to a new home I've also removed the use of htaccess and password files, and done everything I can to ensure that it's not evil.

The new admin page can be accessed at http://cmcowners.comze.com/admin.php. The public test user ID of '0' is still active.

LeThala,

I sent you your username and password via email quite some time ago. I haven't seen any update; did you not get it?