That's because the Mac's "Delete" key is really backspace.
If you keep your files on a network share, you'll have them deleted immediately, even if you'd prefer a recycle bin.
I don't believe there's such a shortcut when using the Finder and I can't remember if there's one if using the Finder replacement (which I much prefer), Path Finder.