Technical on Alkahest

... and logistic regression

Tue, 08 Mar 2016 02:14:50 +0000

A follow-up on the collaborative filter, it occurs to me that it should be possible to add a logistic regression layer in Keras. The following is a regularized multinomial logistic regression model:

from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.regularizers import l2

model = Sequential()
model.add(Dense(numClasses, input_shape=(numFeatures, ), \
    init='zero', W_regularizer=l2(0.01)))
model.add(Activation('softmax'))
model.compile(loss='binary_crossentropy', optimizer='sgd')

As with the collaborative filter, you can easily modify the code to use multiple regularizers and different learning algorithms, say Adamax or adam instead of SGD.

Collaborative filtering in Keras

Mon, 07 Mar 2016 14:56:20 +0000

Ten years ago, Netflix started the Netflix challenge. A contest to see if the community could come up with a movie recommendation approach that beat their own by 10%. One of the primary modeling techniques that came out of the contest was a set of sparse matrix factoring models whose earliest description can be found at Simon Funk’s website. The basic idea is that the actual ratings of movies for each user can be represented by a matrix, say of users on the rows and movies along the columns. We don’t have the full rating matrix, instead, we have a very sparse set of entries. But if we could factor the rating matrix into two separate matrices, say one that was Users by Latent Factors, and one that was Latent Factors by Movies, then we could find the user’s rating for any movie by taking the dot product of the User row and the Movie column.

The Sounds of Science

Thu, 17 Jun 2010 02:07:37 +0000

Last Friday, hsarik pointed out an interesting web site: Echo Nest. They provide a web service that allows you to analyze and remix music. The API also can provide information (meta-data) about music, artists, songs etc. and has Python bindings. If you’ve seen the “More Cowbell” website where you can upload an mp3 and have more cowbell (and more Christopher Walken) added to it, well that site uses Echo Nest and if you download the python bindings for their API, you can see the script that adds the sounds. Personally, I’m fond of “Ob-la-di, Ob-la-da” with 80% cowbell and 20% Christopher Walken.

Model update (updated!, updated again)

Tue, 23 Mar 2010 01:09:45 +0000

Earlier, I posted my current model for predicting the NCAA tournament. Since the whole thing is probabilistic, I figured that I would test it out against the current NCAA standings. I considered four models:

The one that I described
A random selection of which team would win (50/50 chance)
Always picking the top seeded team
A model suggested by a colleague at work

For each model, I ran 10,000 tests and compared them to the current NCAA tournament results, counting the scores for each test. Results are:

They laughed at my theories!

Mon, 22 Mar 2010 19:09:59 +0000

They laughed at my theories. They threw tomatoes when I presented my paper at the academy! Tomatoes I tell you! My minions cower in terror, shrinking in fright from the very ideas contained herein! But I will show them! I will PROVE IT TO THEM ONCE AND FOR ALL. The FOOLS, I WILL DESTROY THEM!! MWAHAHAHAAAA!(ask me how)

Oh, sorry. Where was I? Apparently, there’s this basketball thing going on. Some sort of NCAA tournament that will prove who has the best basketball team. But what if it doesn’t? What if it’s all just arbitrary? Could it be that the chances of any team winning a game are not deterministic, but rather stochastic? I’ll admit that I don’t know that much about basketball. I mean, I played the sport in junior high. I do know the rules. And I even think that it’s a pretty game. But I don’t follow the ins and outs of a particular season.

"Hacking" predator drones

Fri, 18 Dec 2009 18:05:08 +0000

This just makes me sad. Two articles, one in the WSJ, the other on CNN, describing how insurgents in Iraq are hacking predator drones and receiving the video feeds that the drones are sending back to U.S. ground stations. First things first, let’s fix the headlines. Both are running something like “Iraqi insurgents hacked Predator drone feeds.” That should more clearly read: “Iraqi insurgents watching the videos that the Predator drone sends out unencrypted.” Or maybe “Iraqi insurgents watch Predator drone feeds on TV.”

Trash heap of programming history

Sun, 29 Nov 2009 18:08:23 +0000

When we put in the massive wall-eating bookcase, I thought that we had enough book space for the next 10,000 years! Or at least the next 10. Unfortunately, things were filling up a bit too fast, so I grabbed some of the books that I know I’ll never use again and will send them to recycling. Most are completely out of date (from bottom to top, these are from the mid-80s to the late 90s). Linux Application Development is probably still relevant, but a little too basic. Linux Device Drivers goes all the way through the 2.0 kernel series (and may be relevant for the experimental 2.1 series)! Switched LANs (snicker) and In Search of Clusters were given to me by vendors. And if I ever have to program native X-Windows again, I’ll kill myself. I kept the books on C/C++, Python, Perl and PHP - though I’ll probably never buy another programming book. Maybe Celeste is right, I should just get a Safari account… Google works pretty well too. If there’s anything you want in here, let me know… 10258

Great moments in . . .

Thu, 12 Nov 2009 01:39:44 +0000

Minor notes, none worth their own post.

Traffic management: I get a call from K around 5:30. She’s stuck behind an accident and the cops on the scene, a) don’t tell people to take a detour until they’ve been there for a half hour; and b) once the ambulance has left the scene, don’t direct traffic around the one remaining open lane. So, after waiting a half hour, K has to take a 20+ minute detour home.
Memory: Once she gets in, K and I are fixing leftovers for dinner. C: “Hey, where are the mashed potatoes?” K: “Where did you put them?” “In the fridge, but I can’t find them.” “Maybe they’re in the freezer.” “Nope, not there either.” Ten minutes of looking for the potatoes. Did we throw them out on Sunday? Nope, not in the trash. Did C put them in the pantry? Nope. Can’t find ’em, can’t find ’em. Finally, K says, “wait, we fixed rice on Sunday.” There weren’t any potatoes. I would attribute it to getting old, but I’ve always been this way.
FUD (fear, uncertainty and doubt): we’re testing some things at the office – will our authentication system (active directory) honor password failure lockouts when using LDAP authentication? I ask our windows consultant to either a) answer the question, or b) enable an account lockout policy so we can test. He responds back that he can do that, but with the warning that “many Linux services aren’t well-designed for this, and repeatedly try a cached or user-provided password, so that users or service accounts may be mysteriously locked out after one attempt or at some future time when passwords change.” Which is complete and utter B.S. Signs that it’s BS? He references Linux services as opposed to open source, i.e. attempted linux dig. And I used to “own” identity management services, including authentication at a large university and if this was the case, things would have blown up within 10 minutes. I thanked him for the advice and noted that I’ve never seen this, but that it’s why we test.
OS Performance: we’re looking into some new ideas at the office. Things that could be useful as a preprocessor for a host based intrusion detection system. As part of my testing, I told my laptop to audit all syscalls made to the kernel, by all processes on the system. CPU load spiked, system performance went through the floor, the windowing system became almost completely non-responsive. In the two minutes it took to get access to a terminal, I logged 150 MB of audit logs. On the plus side, all of the information we need can be collected. Now I just need to figure out how to keep a usable system.
Self aggrandizement: talking to my technical manager, we need to write up two journal papers based on our recent work. Cool!

I hope everyone had a good Veteran’s Day and remembered to thank the veterans in their lives.

Facebook security vulnerabilities

Thu, 05 Nov 2009 15:32:19 +0000

and this is why I like cross-posting to facebook from my blog. It’s a healthy reminder that nothing on fb is actually private. If it’s online - it’ll be exposed eventually, whether through a new exploit, or just because you “friend” someone in the future that you had written about in the past.

h/t hsarik

Toy!

Wed, 04 Nov 2009 04:04:36 +0000

For a guy who uses computers as much as I do, you would think that I would have better machines at home. But really, no. Our last desktop was eight years old before we replaced it. My laptop was an old Thinkpad X31 - maybe six or seven years old. So, for a combination birthday/Christmas present, I got a new (sort of) laptop. I picked up a refurbished thinkpad X200s from the Lenovo outlet. So far it’s a nice machine: good processor, lots of high speed memory, and a big LED backlit screen. Lenovo sent me notice that they were shipping the laptop today and the next thing I knew, it arrived at the office. The advantages of working 5 miles from the outlet. The only problem is that they didn’t ship all of the parts. I’ve contacted them and they’ll send the dock/dvd.

Dragged kicking and screaming into the century of the fruitbat

Thu, 29 Oct 2009 13:35:41 +0000

With apologies to Terry Prachett, I feel like I being dragged (dragging myself?) into the Century of the Fruitbat. As I mentioned a long time ago, I don’t care for Facebook. I prefer my blog. That said, many of my friends and most of my family use Facebook. So I’m going to start a bit of an experiment. I’ve downloaded and installed the Wordbook plugin for my blog software (wordpress). Starting with this post, in theory anything that’s published on the blog gets cross-posted to fb.

So, this is important

Fri, 28 Aug 2009 03:14:27 +0000

I’m not a big baseball fan. For that matter, there are few ball sports that interest me. But, this is important. If you recall, a few years ago (2004), there was a big furor over steroids in baseball. The government searched BALCO and found evidence of rampant steroid use by baseball players. Now I hadn’t been paying attention to this, but there has been an ongoing legal dispute over that search and how it was conducted.

DDoS-ing good policy

Thu, 27 Aug 2009 02:08:48 +0000

In computer security, one of the most difficult and annoying problems is the distributed denial of service attack (DDoS). The idea behind a DDoS attack is straight forward: the attacker tries to prevent legitimate use of the service by using a large number of other computers. Usually these other computers have been compromised (hacked) and are following the commands of the attacker. Such computers are usually called “zombies.”

There are a number of ways to conduct a DDoS attack, but they are typically variations on the following theme. The attacker instructs the zombies to request access to the service. But the zombies have no intention of actually using the service, instead, they often forge network traffic so that it’s impossible to tell who is making the request. Because the zombies don’t want to use the service, they can make thousands of requests without slowing down. The poor computer hosting the service then sees tens of thousands of requests for access, tries to fulfill the requests and eventually becomes overloaded and dies. The zombies win.

the importance of verifying backups

Tue, 21 Jul 2009 02:58:46 +0000

I was using my personal laptop at a meeting yesterday and grabbed about a gig of files from someone’s usb key. While I was taking minutes, I noticed a lovely new icon that popped up… your hard disk drive is failing. Eeek! Not cool. So, last night I got home and started backing up my files. A few of the new files and a couple unimportant old files didn’t transfer properly. Fine, I could live without them.

Vint Cerf called . . .

Wed, 22 Apr 2009 01:58:23 +0000

… and he and Tim Berners-Lee want you to stop breaking the Internet.

Over the past couple of weeks I’ve had several occasions to be invited into someone’s walled garden on the internet. You know the places. Lovely little sites that are entirely self-contained and which you can’t access unless you are a member? In the old days, Compuserve and AOL were the big walled gardens. These days, it’s Facebook and Linked-In.

blog your type?

Fri, 21 Nov 2008 21:44:47 +0000

This is neat. Typealyzer claims to examine a blog (or presumably any webpage) in order to identify the Myers-Briggs type of the author. It correctly identifies me as an INTP, but doesn’t seem to get etselec.

at home in the (technical) universe

Thu, 20 Nov 2008 15:11:26 +0000

Some recent (somewhat) technical notes:

A while back, I swapped the dead harddrive from my ipod with a compact flash card. Unfortunately, at the time, the biggest (affordable) compact flash was 16 GB, so I lost about half the capacity from my ipod. Not a huge problem, but it became more of one as I added more music. Yesterday, a shiny new 32 GB compact flash arrived and now I’m back to the nominal amount of space on my ipod, except that it’s all solid state and cool. From the technical standpoint, this was something of a PITA, since I didn’t have a windows or mac machine around to reinstall the firmware. My ultimate solution: 1) back up /dev/sdb (boot record and partition table) and /dev/sdb1 (firmware) from the ipod using dd; 2) put the CF in my laptop and format it (a camera would work just as well), this just normalizes the card; 3) put the CF in the ipod (or in the laptop); 4) write the patition table using dd; 5) edit the partition table using fdisk, set the size of sdb2 to be 32 rather than 16 GB; 6) write out the firmware to sdb1; 7) format sdb2 using mkfs.vfat. Voila - a 32 GB ipod CF
If you haven’t seen it already, check out projecteuler.net. They’ve got a bunch of mathematically oriented programming problems online of varying difficulty. Good solutions should all run in 1 minute or less and generally take 100 lines of code or so. It’s a good way to get familiar with a new programming language and to exercise your brain. So far, I’ve done the first 70 or so problems - they don’t take too long, maybe a half hour each on average.
Finally, I got the clutch in my car replaced yesterday. The mechanic said that it was in pretty bad shape and that the (plastic?) bearing the clutch uses had worn completely away. This probably explains why I’ve had no acceleration for the past year (or more?). I had forgotten what it was like to drive a decent car :-)

Computer maps

Thu, 30 Oct 2008 14:17:51 +0000

A few years ago, K was taking some GIS (geographic information systems) classes. That was a lot of fun for me since GIS is something I’ve poked at on and off for quite some time. Back when I was first playing with GIS, GRASS was probably the best (and may still be) open-source GIS system out there, but it wasn’t too user friendly. So it was a lot of fun playing with ESRI’s ArcGIS.

Annotate Flickr

Sun, 05 Oct 2008 02:59:54 +0000

A while back, Luis Villa asked about a script to add creative commons licensing information to an image. I just wrapped up a first cut at a GreaseMonkey script to do exactly that. Hopefully someone will find it useful.

E-book blogging

Wed, 07 May 2008 01:49:39 +0000

I’ve had the Sony Reader now for about a week. In that time, I’ve taken it on a plane trip, read three full books, multiple days worth of the NY Times and I’m in the middle of two books right now. Observations so far:

The electronic paper is very readable. On my plane trip, I must have read for several hours straight with no more eye strain than if I had been reading a paper book. The legibility is good regardless of font size. You might still want to increase the font size if your eyes are tired, but otherwise, there is no need.
The menus and button layouts are pretty reasonable. You can page forward or back. There’s a up-down-left-right cursor that is used to move around on a page. Using the number buttons on the right, you can jump to an arbitrary page in the book. These buttons double as a quick jump to a menu item on the Reader’s standard menus. One gripe, you can only move to different links using up/down on the cursor, left/right don’t do anything. At GB, paideka mentioned that it would be interesting to see what Apple did with the layout and look and feel of a reader. Agreed
Battery life appears to be as advertised: 7,500 page turns per charge. Keep in mind that a page on the reader contains only about half the content of a standard paper back (depending on page layout and font size). Still, around 3,500 pages of paper back text is still pretty good.
Updating the screen is slow. It takes about .5 – .75 seconds to update the screen. A few ramifications: 1) this is almost un-noticeable while reading text; and 2) using the cursor keys is painful, you deal with the update time for each cursor pressed – where ever possible I use the numeric shortcuts.
A third ramification of the slow update time is that the Reader, and almost certainly any other reader using this generation of e-paper, is unusable as a reference book. When I use a reference book, I flip around quite a bit. Forward to the index, back to the text, forward many pages to the next topic, etc. I suppose if the reference book had a really good index, it might be better, but for the most part, this is still not a good tool for referencing which is a real shame.
The bookmarking system is good. Each book keeps your place in the book. The top level of the reader keeps up with the last book you’ve read and your place in that book. You can set any number of bookmarks in each book and then access the bookmarks on a global or a per book basis. It would be nice if the reader also kept a list of most recently read, rather than just the single most recently read book; but that’s a small issue. Typically, I’ll just set a bookmark when I pause in reading, then delete it when I pick the book back up.
PDF conversion still leaves something to be desired. I’ve looked into this a bit. The converter I’m using converts PDF -> HTML -> LRF. The PDF -> HTML conversion uses pdftohtml (surprised?) which is good in some ways, but still leaves off certain things (like images!), at least as used by the reader’s converter. Part of this is due to conceptual differences between PDF and HTML. HTML marks up text, flagging paragraphs, noting images, etc. Ideally, all of this is passed to the browser which handles the layout. PDF will have none of that. PDF consists of a set of primitives that indicate what text (in which font and size) should go in which location on the page. There is no markup of paragraphs, instead, each line of text is described individually. There is no easy way to reconstruct paragraphs from a PDF file (as a research note, I wonder if you could use a partially observable markov decision process?). That said, minus the missing images, the LRF result is definitely readable.

So overall, I’m pretty happy with the reader. The biggest issue is the refresh time on the electronic paper and I hope that will improve over the next couple of years.

Book lover + nerd = ebook

Tue, 29 Apr 2008 14:45:51 +0000

Yesterday, I bought a Sony Reader – the electronic book reader that uses E Ink’s electronic paper. The electronic paper display on the reader is very nice. It uses encapsulated white and black pigments that can be brought to the surface of the page. The only power consumption involved happens when you make a change. Once the change is made, it requires no power to keep the image. The upshot is that you’ve got a very long battery life, a decent contrast ration and a display that can be read in any light – in fact, the more the better since it’s reflective (like paper) rather than backlit (like a monitor). Moreover, because it’s not backlit, it’s easier on the eyes when reading for a long time. I read for a couple of hours last night and it was no different than reading a paper book.

tracks 1.5

Sun, 13 Apr 2008 02:46:29 +0000

Thanks to Luis, I find out last week that Tracks 1.5 has been released. Tracks is the implementation of the “getting things done methodology which I prefer. 1.5 is pretty nice. In particular, I look forward to hiding actions until a particular date. In the past, I’ve wanted to track a todo some six months in the future. I put it on the list and had to watch it for 180+ days.

Web 2.0 and trusting the users

Fri, 11 Apr 2008 13:31:47 +0000

The CTO Project makes an interesting observation that faculty are a bit like some corporations. They feel obligated to use Web 2.0 technologies in order to engage student interest and actually make some token effort to be up to date. But that they only want these technologies if they can exert complete control.

Trust me, I can relate. We’re currently contracting with a part of the government that wants to do something similar. They want to make use of the knowledge of a number of experts to produce an encyclopedia of a given technology. Of course, this has been dubbed the FOOpedia (where FOO is the technology).

Digital Amway

Sat, 29 Mar 2008 02:22:52 +0000

A few years ago, I was accused of using the word “interesting in subtle ways. Sometimes it means a truly novel idea that I would like to learn more about, other times, it’s a novel idea of which I’m more than a little skeptical. In both cases, I stand by the description, to me, both are interesting – but it can make it a little hard to know what I’m really thinking. So take it with a grain of salt that I just read an interesting article in the February 2008 issue of IEEE Computer on how to turn music lovers (particularly teenagers) into music distributors.

Pros and Cons of VoIP

Wed, 19 Mar 2008 14:45:46 +0000

We’ve got voice over IP phones at the office. For the most part, they are a very smart way to handle our communications. We’re too large to want to use individual analog lines through the local telephone company – their rates would be prohibitively expensive. We’re too small to make a private branch exchange (PBX) worthwhile. So sending our voice data over the internet is a good approach. When the phones work, the sound quality is excellent, the cost is relatively inexpensive and the system is fairly reliable. Unfortunately, when the phones don’t work the challenges begin.

FISA extension and telecom amnesty

Tue, 26 Feb 2008 04:10:27 +0000

Few people have been on top of the extension of FISA like Glenn Greenwald. As a quick overview for folks that haven’t been paying attention to the issue:

Late last year a real potential problem with the Foreign Intelligence Surveillance Act (FISA) (as written in the 70s and amended after Sept 11) was recognized. Namely, communication between two foreign entities that was routed through the US was subject to the law’s requirements for a court order. This was never the intent of the law and largely crept in due to the routing of Internet traffic through major US networking hubs
In addition to correcting this, the Whitehouse and the Republican congress pushed for a change to FISA that went beyond correcting the oversight and significantly extended the ability of the government to spy on citizens.
Congress couldn’t pass this permanently, but did pass a six month bill before the August recess, in large part because of scare tactics used by the FBI (releasing warnings of predicted attacks in DC)
Six months was up last week and the Whitehouse was pushing to: a) correct the known oversight, b) extend its ability to spy on US citizens without court order, and now they’ve added c) grant retro-active immunity to the telecommunications companies for illegally helping the government spy both before and after Sept 11th. And of course, if they don’t get all of this, we’ll die in our sleep, murdered by terrorists.
The Senate caved and gave the Whitehouse everything it asked for.
Surprisingly, the House didn’t and we’re now seeing extra pressure claiming that we’ll all die and it’ll be their fault. This is of course BS, but that’s the state of discourse in the country.

I don’t have much to add on the spying per se, but I will admit to being particularly offended and disturbed by the telecom immunity issue. Essentially, these companies started helping law enforcement to monitor calls, read emails, etc. well before September 11th. Their actions were not scared or patriotic, they were largely motivated by greed.

Wordpress title plugin

Wed, 13 Feb 2008 16:01:01 +0000

One of the things I dislike about blogs (and actually most webpages) is that the static content is formatted very nicely using different fonts, etc., but the dynamic content is all pure text. And since there are only a few fonts you will find on every computer, the text all looks pretty much the same: sans or serif. On my own blog, I find that particularly annoying. I’ve got, what I think, is a nice clean looking theme that reminds me of a Victorian-era journal (except perhaps not that fancy). But all of the titles and dynamic content are plain text.

Moving day update

Sun, 10 Feb 2008 20:57:32 +0000

The ISP move went reasonably well. There were a couple of minor glitches. One was a character set problem when I tried to take the most recent database snapshot from Linksky. I’m not entirely certain what happened, but I wound up with extra characters after some punctuation marks. Rather than spend too much time with it, I just grabbed the latest post and dumped it into the Dreamhost site and changed the time.

Moving day

Thu, 07 Feb 2008 04:05:45 +0000

Saturday is moving day. Nope, we haven’t bought a new house. We hope not to have to do that for a long time. It’s time to change ISPs. After only 10 months with Linksky, I’ve had enough. In the past year, between working on the non-profit’s website and my own, I’ve had the following issues:

arbitrary version changes with no notification
random breakage due to “security upgrades – like the time w e couldn’t do anything with a friend’s account or email because his last name contained the letters “curl
random lockouts of service
insults from the security “guru because he thinks he knows more about Unix and IT security than anyone else around
etc.

The last straw for me came earlier this year when a gallery page broke. I tried to login to diagnose the problem and found that they had removed my SSH access. When I emailed, I was to ld there is a “security issue that makes sites with SSH more vulnerable and that they were looking for a workaround and would re-enable SSH by the weekend. Outside of dictionary attacks on passwords, this is really complete BS. Of course, the weekend came and nope, no SSH. Another week rolled around, no SSH. Forget this.

CO2 emissions and compact fluorescent bulbs (updated)

Sat, 02 Feb 2008 19:37:08 +0000

There’s something about the idea of global warming that seems to drive people of a certain mindset completely insane. You start seeing things like: “the planet’s not warming up! “Okay, maybe it is warming, but humans couldn’t possibly be causing it. “Fine. We are causing global warming, but we can’t do anything about it. “What, we can? Well it’ll destroy our economy. “You mean it won’t? Well, you must be French.

Diagnostics

Thu, 31 Jan 2008 02:26:42 +0000

An interesting conversation at work today. I was talking with the guy, L (no – a different one), who has been responsible for our IT for the past 3.5 years. L isn’t the guy that always does the work, although he can. His real job is as an engineer in the company and that’s his primary focus. However, he does handle the technology strategy and monitors the folks that do the IT work. (He and I are talking quite a bit since I’m trying to take some of the load off in this area).

Work blogging

Mon, 28 Jan 2008 03:37:53 +0000

Last week at work was pretty interesting. On Tuesday, L and I flew up to Dayton for the day to meet with our sponsors. That meant a 6am flight out and originally a 11pm return flight. Fortunately, we wrapped it up a bit early and landed home back around 8:30pm. The meeting went well. We were presenting some ideas that I had to the sponsors and they seemed to like them. Strangely enough, in talking to L, he said that he hadn’t really understood what I was proposing until the meeting. Not bad for a guy who wrote a short paper for the sponsors that described what he didn’t understand! But then, I guess that’s why L’s a full professor 🙂

Information markets

Sun, 23 Dec 2007 22:32:53 +0000

One of my projects at work involves information markets: tools to extract aggregate knowledge from groups of people. For example, at Intrade, you can buy and sell contracts on questions like, “who will be the democratic nominee for president? or “how likely is it that the US economy will slip into recession? The market price reflects the cumulative belief that a given event has a certain probability.

IEEE Spectrum also recently did a piece on information markets which talked about Microsoft’s use of a market in 2004 to predict the likelihood of an internal product meeting its production schedule. By opening the market to its employees, who presumably had knowledge of the issues with the product, Microsoft predicted that it would ship three months late – which it did.

It lives

Wed, 05 Dec 2007 04:02:37 +0000

Right before Thanksgiving my iPod went to that great electronics superstore in the sky. More specifically, the cute little 1.8″ hard drive died. Having looked around, I figured the best (or at least most interesting) fix was to replace the drive with a compact flash card. The connector part finally came in today and we had a lovely (and to spare you the suspense, successful) operation.

We started with a 30 GB 5th generation iPod video showing the unhappy ipod symbol:

Rebuilding an iPod

Mon, 26 Nov 2007 04:08:48 +0000

Over the holiday (I hope everyone had a good Thanksgiving!) I spent some time figuring out what to do about my broken iPod hard drive. The simplest/cheapest thing is to replace the drive with another 30 GB 1.8″ drive. The problem is that this is boring. Okay, next thought – up the drive size. I can get a 40 GB drive that’s the same size and is a drop-in replacement. Only problem is that this isn’t very cost effective in terms of dollars per GB. Apparently the wide-spread use of 30 GB drives in iPods has lowered the price point here. What about a 60 GB? It looks like that’s doable too and at a good price point. Downside is that I need a new back to accommodate the thicker drive. Hrm, that’s no good either.

Two factor authentication

Wed, 07 Nov 2007 02:05:42 +0000

A couple of weeks ago, Hunter and I were talking about passwords. More to the point, the inadequacy of passwords and why we haven’t moved beyond them yet. This touches on several points that I made last year. Specifically, that a password that is secure enough starts to restrict its usability.

In a nutshell, authentication is proving that you are who you claim to be. The standard ways of authenticating yourself are through: something you know (e.g., a password), something you have (e.g., a token) or something you are (e.g., biometrics, facial recognition, etc.). So the claim here is that the human brain is not good enough at remembering things to make “something you know secure. Unfortunately, it’s cheap and easy to implement. Two things which are always important.

are there no good ISPs?

Tue, 18 Sep 2007 00:19:49 +0000

I’m starting to think that there aren’t any good Internet service providers. hsarik had troubles with Rimu Hosting. My own ISP seems to be far more focused on (their own version of) security than on usability. At work we’ve been using Pair. I’ve been pretty happy with them until this afternoon around 5pm when our project went dead while we had some 50 people using it.

When we first developed the project, Pair was using php4 (yeah, I know – it’s my own fault for using php 🙂 ). Fortunately, they also provided phpwrap to allow cgi access. Okay, that’s not great, but it at least let our project which required php5 go live. Sometime recently, they made the default php5 without phpwrap. If I had known, well great – but I didn’t see any mention of it. Today, right after I went home they broke phpwrap. Easy enough to fix, but still irritating.

power management and Linux

Fri, 17 Aug 2007 01:12:41 +0000

From skvidal.

If you haven’t seen powertop yet, you’ve got to look into it. Arjan van de Ven, one of the linux kernel hackers and an employee of Intel, released powertop back in May. What is powertop? Think of the Unix “top, but monitoring power, not CPU, usage. It tells you how long you are spending in different CPU states, how long at different CPU frequencies, what is waking up the CPU most frequently, and best of all, it makes recommendations for saving power.

It's official...

Mon, 25 Jun 2007 02:27:02 +0000

K thinks I’m a freak. In her defense, I’m not certain that she would have ordered a $70 keyboard, so she may be right. In my defense, I bought my first IBM-clone in 1990 and have kept the keyboard I chose since then. The company I bought it from, Formosa Computing (for those of you in NC, think Intrex), assembled parts and sold white-box computers. The keyboard that they originally gave me was awful. I didn’t like the mushy keys, so I went back, tried a dozen different keyboards and finally exchanged it for something with a bit more tactile response. With every new computer I bought, I used this keyboard. At one point, it had an AT to PS/2 to USB adapter in order to get it working.

A mental sigh of relief

Mon, 18 Jun 2007 02:40:45 +0000

Working in IT, one of the things that is often on my mind is backing up my data. It’s often on my mind, but I seldom do anything about it. A few years ago, K and I bought an external USB drive to which we occasionally sync our desktop. That made me a bit more comfortable, we’re now covered if the drive crashes. Of course, I then started worrying about a fire in the house. Since my external drive is sitting on top of the computer itself, anything that destroys the one will likely destroy the other.

That's too cool

Tue, 12 Jun 2007 00:46:09 +0000

Okay, thanks to Jag, I’ve seen the coolest thing ever. The steampunk keyboard + monitor:

Now all I need is a few dozen hours of free time to build my own.

Javascript and Form Saves

Wed, 18 Apr 2007 11:06:00 +0000

As I’ve been working on the web front end for the PWC database, we ran into an interesting problem. Our patient records are implemented w/ a “tab style for the display of different parts of the record. Unfortunately, the folks using the database kept tabbing over to a different part of the record which wound up not saving any of the changes they made. My original quick fix for this was to have the tab buttons also save. But that made me nervous. I worried that if someone accidentally changed things or if I was giving a demo, a change to a new tab would save bad data.

It's quiet, too quiet...

Fri, 13 Apr 2007 14:26:56 +0000

I haven’t been so much with the blogging lately. For me, that is generally a sign that things are busy enough that I’m living life rather than blogging it [ed note: For all of you young readers, this is what is known as irony. When I have the most to say, I have the least time to say it in; conversely, when I write the most, I have the least to say].

Copyright infringement on campus

Mon, 02 Apr 2007 15:33:05 +0000

I can’t turn around anymore without hearing something new about how colleges and universities are public enemy number 1 with the copyright industry. In early February, the Recording Industry Association of America released a top 25 list of the schools to which they have sent notices of alleged copyright infringement. Right after that, they started announced that they would start suing students and that they were going to send “pre-subpoena and “settlement notices to campuses.

Wolverines!!

Wed, 28 Mar 2007 01:03:49 +0000

er, sorry just a minor Red Dawn reference – it’s not my fault, I grew up in the 80s.

Now that I’ve got that out of my system, a note on a new toy. As I mentioned before, I’m going to go (mostly) digital on my trip to Yellowstone. I bought 8 GB of memory cards which is only 800 or so pictures in raw format. For the past several years, I’ve shot around 800 pictures in film. On the one hand, I suspect I’ll take more pictures in digital since the marginal cost is basically zero. On the other hand, I’ll have the ability to prune out any bad shots. Still, I suspect that I’ll wind up wanting more memory cards.

John W. Backus dies

Thu, 22 Mar 2007 02:45:43 +0000

Back when I was a postdoc, I was studying FFTs. Okay, actually, I’m not enough of a mathematician to study FFTs, I was studying the efficient implementation of FFTs on parallel computational clusters. One of the graduate students I was working with on the project penned the following:

If there’s a working FFT That’s not in FORTRAN but in C Alas it twice as slow will go And that (as always) goes to show If speed is your greatest asset, Then what you C is what you get. -V.V. (Mony) Iyers

and people wonder why the change to DST worries me...

Wed, 28 Feb 2007 23:38:06 +0000

In 2005, congress mandated a change to daylight savings time, essentially, starting it three weeks earlier (March 11th, this year) and ending it three weeks later. Our local paper is requesting suggestions for what people will do with their 22 extra hours of daylight. Here’s my suggestion: spend the time fixing all of the computer problems caused by the DST change.

Essentially, a change to DST is very similar to a self-inflicted Y2K problem. If you want to get a sense of how time/date issues can affect computer systems that aren’t prepared, take a look at this article regarding the new F-22 Raptor and it’s problems with the International Date Line. I wasn’t worried about Y2K because we knew about the issue for years and most modern operating systems and software had already addressed it. With the DST changes, we haven’t had nearly enough notice or preparation. Now, I’m not stocking up on canned goods, but I’m pretty certain that March 11th is going to bring about extra work.

bah! - updated

Tue, 02 Jan 2007 01:20:26 +0000

so Saturday night around 11:30pm, the acid neutralizer I just installed, exploded. Okay, it didn’t really explode, but the top popped off again. Spoke to the customer support, looked around a bit more and it seems that the threads on the tank were stripped. I’ll call tomorrow and see about getting a replacement tank

Update: I just received a call from the folks that sold me the acid neutralizer (I didn’t even have to call them back). Two interesting things: 1) apparently, the tank head they sent is a down flow head and the system I’ve got is designed to be upflow. Not a big deal, except that I have to reverse the inlet and outlet plumb lines as compared to what’s labeled on the head. Normally they send a note along those lines – this time they didn’t. 2) Given that one of the threads is stripped, they’re going to send me a new tank and an RMA to return the old one.

Web related notes

Sun, 31 Dec 2006 03:19:32 +0000

Two web-related notes:

I’ve just migrated the Piedmont Wildlife Center’s website from a bunch of php webpages (the only reason for the php was to include the template) to the Drupal content management system. I’ve also added in the Coppermine photo gallery to keep our pictures. I would have used gallery2, but it kept dying due to memory errors with the database. I’m guessing our ISP has the ulimit set to small. Overall, the updates went pretty well, the biggest headaches were taking us out of a table based layout into the (more) modern world of CSS. I kept the current look and feel, but the CMS should make it easier to change in the future. But for now, at least I can distribute the web updating responsibilities to others 🙂
Does anyone know how to get WordPress to do better detection/blocking of comment spam? It’s killing me. Fine, I’ll go google for it.

New books

Sun, 17 Dec 2006 22:56:11 +0000

I had a chance last week to read a few books, all of which I would recommend in some fashion or another:

National Geographic’s “The Ultimate Field Guide to Photography This is a very good, basic field guide to photography. It covers a huge range of subjects and, not surprisingly, has some beautiful photographs. The book covers both film and digital photography, editing, archiving, composition, and has a great inspirational chapter describing Robert Clark’s photographic travelogue using only the camera in a cell phone. My only complaint about the book is that it covers too many topics, but not much in depth. It’s a great book for a beginner, but there wasn’t too much I hadn’t seen already.

predictions... "Cyber Monday"

Mon, 27 Nov 2006 16:40:30 +0000

Let me get a prediction in here before the data come out tonight or tomorrow. “Cyber Monday will be a disappointment to retailers. They will do more electronic business than they did last year, but not significantly more than they did last Friday or next Monday. Any uptick is going to be small and due to people paying extra attention to online sales.

“Cyber Monday (sorry, I can’t help but put it in quotes) is yet another marketing ploy. The theory is that when people go back to the office, they’ll use the high-speed connection to shop online. The problem is that the majority of people who shop online already have a high-speed connection at home. Add to that the companies that are cracking down on online shopping (and game playing and general web surfing) from the office and I’m guessing that it’ll be a non-event.

Poindexter

Fri, 24 Nov 2006 17:55:40 +0000

It’s taken me a bit to write about Admiral Poindexter’s visit and the small group talk we had with him. Let me start by reminding folks that here’s a guy who was convicted of lying to congress. The conviction was later overturned on a technicality. He’s also very politically savvy. I once asked my father if he would ever pursue becoming a general in the army. He told me that he was hoping to make full colonel (he later retired as a lt. colonel), but that becoming a general required a literal act of congress and that you needed to become a politician. I would assume the same thing is the case with an admiral and doubly so in the case of Poindexter who managed to become the highest ranking geek in government. All of which is to say take my impressions with a grain of salt.

/me grumbles

Thu, 16 Nov 2006 02:20:31 +0000

bah – the past few nights, my website/blog keeps getting screwed up because all of the disk space on the drive is consumed. the first night, i freed up some space, but didn’t pay attention to how much was free. it happened last night and again i freed up space, noting that i had about 700 MB free. Tonight, it happened again and I finally tracked down what happened. I do an rsync backup to an external drive every day. A couple of days ago, I rebooted the machine and the external drive didn’t mount back up. So I wound up doing a backup of my home directory and root partition to the root partition. Of course the partition filled up. Oh well, problem solved now. I just need to be more careful in the future

North Korean nuclear test

Tue, 07 Nov 2006 00:03:14 +0000

Well, it seems that we finally know what happened with the North Korean nuclear test that fizzled. They apparently mistranslated the Arabic documents the U.S. posted online.

Okay, so neither of those is really very funny. On the one hand, the U.S. posted a whole host of Arabic documents from Iraq that had never been examined before in the vague hope that someone would be able to find evidence that Iraq had a WMD program before we invaded. This was idiotic. It’s equivalent to my posting an entire database of personal information in the hopes that someone online could determine if there were social security numbers in it.

Biometrics - fingerprint scanners

Mon, 23 Oct 2006 14:00:05 +0000

I recently had a small argument with a vendor selling biometric fingerprint scanners tied to your credit card number. He said that they were the greatest and most secure thing ever; I said that there weren’t any standards and that the security of the devices was questionable.

I wish I had seen this earlier.

Things that make you go blind

Tue, 17 Oct 2006 20:42:07 +0000

No, not that – Programming.

I need to do some statistical work to test a theory. The best approach in order to compare my results is to do the work in R. But it looks like I’ll need to extend a package. I download the package from CRAN (the Comprehensive R Archive Network), unpack it and start trying to figure out how it all works. Unfortunately, it is written in Fortran, C and R. Untangling it will take forever.

Thinking about security and usability

Sat, 14 Oct 2006 03:06:24 +0000

IT security (and for that matter, other security concerns too) are often seen as conflicting with usability. There is something to that. If you take any given technology and turn up the level of security it provides, you will almost always decrease the usability of the system.

Consider passwords. If people are allowed to choose their own passwords, they will typically choose something very usable for them. They’ll pick their dog’s name, their wife’s name, their userid, etc. These passwords don’t provide much security. To compensate, we often turn up the security knob and require “stronger passwords, e.g., minimum of six characters with no dictionary words and multiple “character classes.

Stock spam

Sun, 24 Sep 2006 22:26:26 +0000

One of the disadvantages of having so many email accounts is the number of spam you get. Recently, I’ve been noticing an increase in stock spam making it through my spam filters. I’ve been wondering how effective the spam is and whether or not one could make money shorting these stocks.

Apparently, I’m not the only one. The local paper carried a NYT article titled “ Many people fall for stock spam. In the article, the author describes the work of Frieder and Zittrain. Frieder and Zittrain found that pink sheet stocks that were heavily touted in spam were significantly more likely to be traded than non-touted stocks. Purchasing these stocks would lead a 5.25% loss within two days. For the most heavily touted stocks, the average loss was almost 8% in two days.

carrier thermidistat design - let's put it in the category of needs to be fixed

Wed, 23 Aug 2006 02:56:16 +0000

Back when I was a grad student and just about to finish up, the school of engineering turned a class of 60 freshman over to me to teach intro to numerical methods for engineers. I tried to liven things up by emailing the occassional engineering funny to my students who even reciprocated. The following is a brief excerpt from an engineer identification test that was sent to me by a student.

things that make a security officer cry

Mon, 14 Aug 2006 19:33:35 +0000

I spent a lot of time last week looking at an application in order to assess its security. The thing that was troubling me was that this is a web application and the primary form for data entry was defined like:

form name=foo method=post action=

This means that the nothing happens when you hit submit on the form – at least not in the html world. So, I took a closer look and found that each of the buttons (submit and clear) actually had a field “onclick=’doSomething();’ attribute.

High dynamic range photography

Fri, 04 Aug 2006 02:59:42 +0000

Okay, now I’m just embarassed. I just heard about high dynamic range (HDR) photography from a NY Times article. I plead that it’s mostly just a trick and, hey, I primarily shoot slide film anyway.

So for others who haven’t heard of HDR photography, it’s a technical work around for one of the more difficult photographic problems – the problem of the dynamic range of film vs that of the eye. In a nutshell, the human eye has an enormous dynamic range. It can distinguish between two different very bright shades while simultaneously distinguishing between two very dark shades. It’s dynamic range, IIRC, is somewhere around 10. On the other hand, slide film and prints from print film have a dynamic range of 5, maybe 6. That doesn’t sound like much difference, except that this is a logarithmic scale.