Okay.
I have made that same mistake and had similar symptoms as what you are describing. Just to be clear you should use the Image found in:
buildroot/output/build/linux-custom/arch/arm/boot/Image
Are you sure that you are compiling the kernel to boot from a ramdisk? I think when I did this I may have had to do a make menuconfig in the linux source directory to set this configuration. I was only ever able to get this to work by setting the option to boot from a ramdisk appended to the binary. I've attached my defconfigs from when I last did this.
- John