As you have an '84, the bumper pad cars have a unique upper mounting bar that slips under the taillights. It is unique for bumper pad cars, but the same for all other style bumpers that were used (aero notchback, aero coupe ['87-'88], and fastback GT).
As mentioned, what you actually need is the notchback ribbed bumpers. They are different than the fastback GT bumpers. They look the same at first glance, but you need to look at more photographs to determine the different. The fastback GT rear fascia has rounded moldings and a "GT" emblem on the lower right. On the ribbed molding cars ('84 indy, '85 GT, and '86 SE), the bumper has no GT markings but has 'PONTIAC' molded into the center of the fascia. It will look similar to what you have on your car now, but overall look like a GT fascia. If you attempt to use an actual fastback GT rear fascia you're going to run into the issue of the fascia not having the same molding.
Here's the fascia I'm talking about for notchbacks. Notice it looks like a GT fastback fascia, but it's not.

Here's the fastback GT fascia. See the difference?

They actually look near exactly the same at first glance, and are actually very close in overall shape, but there is differences.