All LPRs use their green cards instead of passport at the GE kiosk. You found out what happens when you try to use your passport instead

You were probably inserting the greencard the wrong way. Machine readable code (back bottom side of the card) has to be in the same way that it is on a passport. But card definitely has to be face up (IIRC, with the top of the card towards you, but its been a while, so I could be wrong).

Yes, there is always a manual entry button on the screen as well you can choose.