From the NJ E-Z Pass Customer Reference Guide:

"TERMINATION
You may terminate this Agreement at any time by returning all of your E-ZPass
tag(s) to us in good working condition except under normal wear. Tags can be
returned in person to any of the New Jersey E-ZPass Walk-in Center locations or
by mail to the New Jersey E-ZPass Customer Service Center. Customers damag-
ing or defacing a tag through misuse or abuse will be required to pay the full cost
of the tag (minus any tag deposit). Such damage includes, but is not limited to,
cracks, cuts, or gouges and/or pencil, pen, permanent marker, stickers, and any
other writing or marking whatsoever.
Tag(s) will remain New Jersey E-ZPass property under all circumstances. Upon
termination and return of your tag(s), your account balance and tag deposit will
be refunded to you. All outstanding charges will be deducted prior to refund. You
agree that we may terminate your E-ZPass account for improper use at any time,
and without notice."

Yes, you can get an E-Z Pass transponder in NY and use it anywhere E-Z Pass is accepted. But a NJ-issued E-Z Pass transponder will give you a discount on NJ tolls; a NY-issued transponder will not.

A "tag" and a "transponder" are the same thing when we're talking about E-Z Pass.

Wherever you get your transponder, it would be best if you provide a valid credit card for auto-replenishment, so that you do not run out of money on your E-Z Pass account.