You definitely won't be able to use a Group in a boolean operation. Only paths, although surprisingly, ellipse/circle, rectangle/square, or star/polygon shapes can be used.
When you say you want it to look like a stamp, do you mean a postage stamp? Or do you mean a rubber stamp, where you press it on a pad of ink, and then press it on the paper?
If you mean rubber stamp, and you want all those white flakes or chips to be cut out of the text, you need to select all of them, and do Path menu > Combine. And with the text, you need to do Path menu > Union. Then select the text and the rectangle, and do Path menu > Combine. Also, as theozh suggested, watch the status bar. If something doesn't work, it will probably be telling you why.
After that the Difference should work - when you have 2 paths and only 2 paths selected.
I know an old tutorial which can help with the rubber stamp:
http://howto.nicubunu.ro/rubber_stamp_inkscape/Same author has one for postage stamps:
http://howto.nicubunu.ro/inkscape_postage_stamps/