|
September 30th, 2009
08:16 am - Tricky Rails POST error Hi all,
I ran into a tricky Rails issue yesterday and wanted to share the solution.
I had a form along the lines of the following:
<form action="/example/myaction" method="POST">
<input type="hidden" name="object" value="123">
<input type="text" name="object[name]">
<input type="submit">
</form>
Clicking the "submit" button generated the following unhelpful exception:
Conflicting types for parameter containers. Expected an instance of Hash, but found an instance of String.
This can be caused by passing Array and Hash based paramters qs[]=value&qs[key]=value. What does THAT mean? As it turns out, Rails iterates through the posted variables and puts them into the params hash. With the example above, that means that it would try to create a hash named params[:object] (so my text input value would be addressable at params[:object]['name']) and also a string named params[:object] (for the hidden input). Since params[:object] cannot be both a hash and a string, the exception occurs. The fix is simple: change the name on one or the other. The following form works:
<form action="/example/myaction" method="POST">
<input type="hidden" name="object_id" value="123">
<input type="text" name="object[name]">
<input type="submit">
</form>
The Google was pretty weak in explaining the source of the exception, so maybe this will clear things up for someone, somewhere.
|
August 11th, 2009
08:52 am - Ads on web pages I use AdBlock Plus most of the time. If I really like a site, I will disable AdBlock for it so I see its ads. Most of surfing, though, is an ad-free experience.
Today, I went to the CNN website with a browser that wasn't blocking ads or popups. To my shock and dismay, they are using pop-under ads! You know, ads that pop up in their own window, then immediately shuffle back to hide behind the window you're on? Holy cow! Am I the only person who considers pop-up ads to be in poor taste, and pop-unders to be insulting as well? Would you ever buy something that was advertised on a pop-under?
|
July 1st, 2009
07:42 am - AdSense and consumer-generated content I got an e-mail this morning from Google AdSense. It was a "prohibited content" warning, as one of the hundreds of thousands of users on my site decided to post porn to their podcast. While it was a simple matter to disable the podcast they named, I'm irritated by two things: firstly, they state that there may be other violations they didn't tell me about and which I am still responsible for, and secondly, their policy is too vague to be enforced using software (hence the absence of a tool which tells you whether you violate their policy). Great-- so all I have to do is look at each of the millions of pages on my site, every day, and check against Google's laundry list of prohibited items. Then, if I'm lucky, I will catch violations before Google does. If I'm not lucky, Google will interpret their own rules differently than I do, or someone will post a violation after I check and before Google does, or my head will explode because I'm reading 34 web pages per second and trying to determine whether two of them are mostly Hungarian or Bulgarian.
|
April 20th, 2009
09:05 am - Same as it ever was Seeing this whine about progressive voting power got me thinking this morning about the younger generations.
I used to agree with the OP, but now, I've changed my mind. Actually, I changed my mind once I got to Oklahoma and realized that, if anything, the younger generation is more conservative than that of our parents. Remember the "Bong hits 4 Jesus" case?The First Amendment is already in trouble with the rising generation. A 2006 survey by the John S. and James L. Knight Foundation found that 46 percent of high school students think newspapers should only publish stories with government approval.
After that disillusionment, I hoped that what we were seeing was an increase in variety-- that perhaps, although they're not moving left, they could be moving in both directions. Seeing stories in the paper about those who reject the economic principle of scarcity ("Scarcity is a lie propogated by those who want us to accept the idea of property, which (our not possessing) is in turn used to keep us from power.") and ultra-conservative kids' groups like the Jesus Campers could lead to such a conclusion. To test the hypothesis, I considered the idea in the context of my parents' generation.
My parents were born among the earliest (oldest) boomers. They were in their 20s during the 60s. They were born before the Vietnam War started and were voting before it was over (and would have been voting before it began, but there were still property requirements and the twenty-sixth amendment wasn't adopted until 1971). They spent most of their lives under the blanket of fear called the Cold War. Nothing like the Cold War to make one more conservative, eh? In this environs (remember McCarthyism? That happened when my parents were kids.) it's important to remember that two of the three issues mentioned above (drug legalization, gay rights) made huge strides toward acceptance. Marijuana was very nearly legalized under Carter, and gay rights went from a joke to law thanks largely to the Boomers (and, believe it or not, their parents). The Boomers are rife with progressive activists.
What happened to all the progressives of our parents' generation, then? Nothing-- they're still there, and they're still making a difference. Two factors are working against them, though. First, being rich seems to result in a rightward move in the political spectrum for many, and second, having money seems to help one stay healthy and alive longer. As a result, old people are, as a group, more conservative than the same group would have been 20 years previously. That's all-- we're not witnessing a revolution; it's just demographics.
Based on that, what about this generation? Some of the soldiers in Iraq and Afghanistan were born after the start of Operation Desert Shield. For them, Iraq is our U.S.S.R. and the Cold War is not so cold--their high school classmates are out there dying from IEDs. The conservatism of the younger generation far exceeds that of our parents'.
I don't believe for a moment that we'll see any significant leftward move as the older generation dies. If anything, we'll see a rightward move as the younger generation-- the children of the Iraq War--come of age.
|
March 12th, 2009
04:25 pm - Fun with Ruby's C String hashing function I've been hacking on the Ruby 1.8.6 source code today, trying to figure out why the string hashing function returns different values on 32-bit vs. 64-bit platforms. I'm pretty sure it is due to different byte-widths on data types used, but my investigation indicates that the widths used are the same. What's more, when I pull the algorithm out of the ruby string.c and compile it as a freestanding program, I get the same output on both 32-bit and 64-bit platforms! Augh. Now I'm digging into the compile-time options to see if they have any effect. For the curious, here's my simplified version of the algorithm (which runs the same everywhere):
#include <stdio.h>
#include <string.h>
int str_hash() {
char str[] = "errorvaljustin";
int len = strlen(str);
char *p = str;
int key = 0;
while (len--) {
key = key*65599 + *p;
p++;
}
key = key + (key>>5);
printf("Key is %i\n", key);
return(0);
}
I'm halfway there. Even though I don't yet know why it does so, I have a hashing function that returns the same thing on 32-bit as 64-bit platforms. My eventual goal, though, is to write a hashing function that returns the same value as the 32-bit platform's built-into-ruby String#hash, regardless of platform.
|
March 10th, 2009
01:29 pm - The strange language of find(1) I had a disk filling up with backups. The question: What files are on the disk that were not created by the backup software (maatkit)?
The one-liner to figure it out:
find . -name "*.gz" -prune -o -name "00_master_data.sql" -prune -o -type d -prune -o -print
What does that do? Most people don't get this involved with find(1). The secret sauce, in this case, is the -prune idiom. It's a bit hard to understand until you start viewing the arguments (after the first) as a boolean expression to be evaluated for each file found. -prune means "print to /dev/null" if that helps. In plain English, then, the above command would read, "Look through this directory and all its subdirectories; if a file matches "*.gz" then print its name to /dev/null, or if a file matches "00_master_data.sql" print the name to dev null, or if it is a directory, print its name to /dev/null, or print the name to STDOUT.
I built this command iteratively, so it is not optimal, but I'm choosing to leave it as written instead of editing it because the original above shows a bit more of the thought process that went into its composition.
|
March 8th, 2009
06:35 pm - using mv with xargs I'm currently moving a bunch of normalized directories (as in, the names are shell-safe) into a temp folder. The list of directories is in a file (so I can remember the original locations of the directories easily if I have to put them back).
When the list of files to move is large, the semantics of the mv command are slightly different. Something like mv `cat list-of-files` /tmp will fail with Argument list too long or equivalent.
You could say
while read line ; do mv $line /tmp ; done <list-of-files
But that runs mv once for each line, incurring significant overhead. xargs was made for this purpose, but by default, it tosses a pile of arguments onto the END of the commandline, e.g. xargs foo < list-of-files.txt will run foo line1 line2 line3 line4... until the file is exhausted.
Mike Parker, David MacKenzie, and Jim Meyering thought of this and added an option that comes in handy when combined with xargs. Now that I know about it, I can do
xargs mv -t /tmp < list-of-files
and I'm done!
|
February 6th, 2009
09:54 am - Fun with transferring files: ssh, tar, and gateway servers I had an interesting time yesterday with transferring a database backup from my backup server to a development server for testing.
The scenario:
My backup server is a linux box on a private network. To log into it, I have to pass through a machine in a gateway cluster. All logins are over SSH. A typical command for logging into it would be:
ssh -t gateway-server.my.domain ssh backup-server
I use the excellent Maatkit, specifically mk-parallel-dump, to back up my database. It creates a directory containing one file per database table in the set.
A good way to transfer files over SSH when you're starting from scratch (i.e. no older versions of the files on the receiving end) is to use tar. For example:
ssh some-server tar cf - mydir |tar xf -
would transfer mydir from some-server to the local box. Handy. In this case, though, there was a wrinkle-- the gateway server. Adding that inflection gives us:
ssh -t gateway-server.my.domain "ssh backup-server tar cf - mybackup" | tar xf -
If you're using password-based authentication, you'll experience some strange behavior here. The first password prompt (for gateway-server) will show up, but the second will not. Also, tar will complain:
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Error exit delayed from previous errors
The complaint springs from the fact that the password prompt from backup-server is finding its way into the pipe to tar. Why is this? The Unix convention is to send prompts and error messages to stderr, specifically to prevent such confusion from taking place. The shell's pipe only accepts input from stdout, so you see the prompt, enter the password on stdin, and all is well. When you add the second ssh command, though, stderr and stdout from the second ssh are merged into stdout by the first ssh, as the first ssh "owns" stderr. As a result, your tar output has the string Password: ^M prepended to it (assuming you typed the password correctly). The second password input must be made blind--you won't see Password: because it's getting slurped up by the piped command at the end. The ^M is the remote echo of the Enter key you pressed after you typed your password.
To silence tar, you have to strip out the extra crap before it gets there. Unix command-line tools to the rescue again:
ssh -t gateway-server.my.domain "ssh backup-server tar cf - mybackup" |dd bs=12 skip=1 |tar xf -
What happened there? dd is a command that reads stuff from one place and writes it somewhere else. If you don't tell it where to read from, it picks stdin, and if you don't tell it where to write to, it picks stdout. The bs=12 part says "One block of data is 12 bytes." 12 bytes is the length of the string Password: , plus ^M and a newline. skip=1 means "skip the first block." So, "dd bs=12 skip=1" means "read from stdin in blocks of 12 bytes and write all but the first block to stdout." tar then picks up data from stdin and writes a directory.
|
January 9th, 2009
09:00 am - Giving up on Microsoft Despite the fact that I've been running, on and off, Linux on the desktop for the last fifteen years or so, my primary work machine at Podomatic ran Windows XP.
It made me feel a bit like a traitor, honestly. I owe my career and a large part of my outlook on the world to Linux, GNU, and the free (as in speech) community. All that aside, I couldn't justify the time it would take to install Linux on my (pre-installed with Windows) PC and then get everything working to my expectations.
( Read more... ) Current Mood: pleased
|
June 17th, 2008
04:36 pm - Informative post on the 5th Amendment
Make no statement to the police under any circumstances. - Supreme Court Justice Robert Jackson
Explanation of why
|
10:40 am - On brain-dead commercial software and open source fixing Occasionally, I find myself writing software based on open-source fu to work around a stupid and/or annoying bug in a commercial package.
The stupid, annoying bug of the past few weeks has been with Adobe Flash Media Streaming Server 3. Overall, it's a pretty solid piece of software. It costs $1000 for a license (and nothing else; support starts at $500 per incident and it's another $3500 to actually write software for it). It has one annoying bug though-- it mishandles id3v2 tags. I suspect that they're mis-parsing the id3v2 header in the first ten bytes of the file, because the bug gets worse as the header gets bigger.
iTunes will display an image for an mp3 if you embed the image in the id3v2 tag for the mp3 file. Pretty cool. That also grows the id3v2 tag to about 45k. This amplifies the bug (which Adobe has yet even to acknowledge, let alone fix). The bug manifests by starting RTMP streams late. In layman's terms, that translates to "the player skips the first 2-10 seconds of the media being played, and there is no way to get back to hear it."
Given that we needed to keep the image in the id3v2 tag for iTunes, but we couldn't allow the truncation to continue, we were planning to make an additional copy of every single mp3 on our site and serve it for playback. That would take a ton of storage space, time to convert (hundreds of thousands of mp3 files), and overall it would suck.
Enter open source. "Let's write a filesystem overlay to trim the id3v2 tags from the media on read!" That way, we don't have to rewrite any files, and the tags are up front (therefore, conquered with a simple seek()). Within two hours, we had downloaded and installed FUSE for linux (Filesystem User-Space Extension), FuseFS (ruby FUSE implementation), and an initial implementation of the idea was running in dev. Today, it's out in prod and working perfectly.
The only issue we had with it was that the API required files to be read, then passed in memory. That caused our FuseFS implementation to use tons of memory (many of these mp3 files are >100 MB). One email to the author garnered a quick response with very helpful advice on how to pass a descriptor instead. Adobe can take their $500-per-call support center and shove it. Open Source saved the day. Current Mood: triumphant
|
April 16th, 2008
01:55 pm - How I fixed my overheating router At work, I am responsible for a Netgear WGR614 router. Overall, it's a good value-- the thing was only $20 three years ago, and it hasn't broken (much). After a couple of years, though, the bottom plate started yellowing from heat. The broadcom chip that does all the heavy lifting had no heat sink!
Because of excessive heat (my theory), the dang router would interrupt the wireless signal every few minutes and then recover on its own. I'm imagining some sort of DMA reset happening in there. Anyway, it was clear that the router needed a heat sink. Not about to spend $10 on a little memory or chipset heat sink, and then another $15 for special heat sink epoxy (remember, that would be more than the value of the router!), I improvised.
Five days later, no drops. Yes, that's a PCI slot cover, and yes those are wire ties. I had a tube of heat sink compound sitting on my desk (don't we all). Current Mood: accomplished
|
March 29th, 2008
01:23 pm - sed(1) to the rescue...again Frequently, I find myself impressed with the decades-old Unix stream editor, sed. It took me a while to learn it to the expert level, but it's (very) worth the trouble. Though I could have written a longer program in perl or ruby to do the same, sed's convenience for one-liners makes it easy to rattle off single-use regexp piles for easy changes.
One thing I do a lot is compare two directories, e.g. /home1/user and /home2/user. Sed to the rescue: diff -r --brief $(pwd) $(pwd |sed 's/1/2/') Today's use was a bit more complex. ( Read more... )
|
February 26th, 2008
09:39 am - Postfix, MySQL, and stored procedures Yesterday, I discoverted that Postfix's MySQL aliases support doesn't support stored procedures, or any query that returns a resultset instead of a single result. Note that "select 1" returns a resultset.
Unhappy with that limitation, I wrote a quick and dirty patch. (I'll post this to the Postfix devel list in a bit.) It works with the two versions I've tested, 2.3.6 and 2.4.5. I get the feeling that the mysql dict support is rather stale.
To use, just change mysql-aliases.cf (or whatever your alias file is) to say
query=CALL my_proc('%u') # or whatever it is
instead of
query=SELECT ... WHERE ... = '%u'
( Read more... )
|
February 3rd, 2008
08:42 am - Database queries I believe I've read this blog post by Sergey Petrunia a couple of dozen times this past week. Understanding how indexes, ORDER BY, WHERE, HAVING, and GROUP BY all interact in the database server (MySQL in my case) is not easy.
The sentence that has become my mantra: [Indexes] can be used when the first non-const table in the join order has an index that matches the ORDER BY list. The good news is that all this query optimization has given me a deeper understanding of our dataset and how it is growing., and I'll know how to avoid adding too many indexes to tables as I develop.
|
January 25th, 2008
02:23 pm - Am I the only one who does math in my head like this? Quick, what's 143.4 times 14?
143.4 * 14 = (14 * 100) + (43 * 10) + (4 * 40) + (4 * 3) + (0.4 * 4) + (0.4 * 10)
Other days, I would just do
143.4 * 14 = (143.4 * 10) + (4 * 100) + (4 * 40) + (4 * 3) +(4 * 0.4)
In grade school, they called this "chunking" and I ignored them.
|
01:30 pm - Not feeling fired up? The Administration has been in a full-court press to bully Congress into making horrible permanent changes to FISA -- including immunity for telecommunications carriers like AT&T -- based on the argument that critical surveillance of terrorists will be cut short or degraded if the Protect America Act (PAA) expires on January 31, 2008.
But no surveillance started by the PAA will end when the PAA expires. All of the spying done under the PAA will continue until at least July 31, 2008 even if the law goes to the dustbin of history on January 31, as it should.
Congress Stand Firm: Surveillance Continues Even If PAA Expires
|
January 24th, 2008
12:18 pm - Averages One bit of nerdy coding I've clung to since college is sample-based average calculation. Some call this a "running average" though the term is a bit overloaded.
Typically, one calculates the average of the numbers in a list (i1, i2,…iN) as the sum of i from 1 to N, divided by N. On computers, this runs into trouble when there is a very large number of samples, either because N has exceeded the system maximum limit or, more commonly, because the sum of i from 1 to N exceeds that limit.
Recent computer languages do extra work to prevent running out of numbers, and that is why the following is a guilty pleasure. I've coded this in Haskell, C, Perl, Ruby, Java, HP RPL, and probably a few other languages.
my_array = [1,2,3,4,5,6,7,8,9,10]
samples = 0
average = my_array.inject(0.0) do |avg, sample|
samples += 1
avg + (Float(sample) - avg)/samples
end
More succinctly, betraying my inner C programmer (ruby doesn't have ++):
samples=0
my_array=[1,2,3,4,5,6,7,8,9,10]
average = my_array.inject(0.0) {|avg, s| avg + (s.to_f-avg)/(samples += 1) }
That's really not much more difficult than
my_array=[1,2,3,4,5,6,7,8,9,10]
average = my_array.inject {|sum, s| sum + s } / my_array.length
and it works until you have more samples than MAXINT, whatever that is in your language (sadly, Bignum in Ruby is conceptually infinite).
|
January 18th, 2008
11:25 am - Memoizing and transactional fixtures I just upgraded a Rails 1.1.6 app to Rails 1.2.6, and one of my more complex unit tests failed. As it turns out, 1.2.6 has transactional fixtures, and 1.1.6 does not. I had get_foo methods that would memoize up to 1000 records as they were inserted into the tables, and they work perfectly. Unfortunately, though, Rails was running a ROLLBACK on the db each time a unit test would run, then calling setup again (as opposed to running the teardown method without a rollback). After the ROLLBACK, the memoized methods continued returning their saved records, but the records no longer reflected what was in the database.
The fix?
self.use_transactional_fixtures = false
|
January 15th, 2008
12:55 pm - Fun little bit of ruby code for Rails With ever-growing tables in a production environment, doing stuff to all rows in a table becomes a memory-stretching endeavor.
Code from the console like
>> User.find(:all).find_all {|u| u.has_spork? }
can exhaust available memory in a heartbeat. However, you still need to find the sporks, as it were, and the following bit of code can make them appear with much less memory use:
class Chunkable
# allow a method to process 1000 rows at a time instead of loading the entire
# table into memory
def self.chunk(klass, num_items=1000, opts={}, &block)
item_count = klass.count
current=0
while current < item_count
items=klass.find(:all, {:limit => num_items, :offset => current}.merge(opts))
current += num_items + 1
items.each do |item|
begin
block.call(item)
rescue Exception => e
raise e
end
end
end
end
end
The example above then becomes
>> found=[];Chunkable.chunk(User){|u| found << u if u.has_spork? };found You'll still run out of memory if you find too many records, but at least the act of searching won't kill you.
|
|
|
|