Yes you have to pull the panel off. Be careful because there are tabs and xmas tree fasteners. Trees are replaceable but the tabs are not if you break them. Just pull them out enough to shine a flash-light inside so you can locate the fasteners, then you will be able to figure out how to remove them. I wanted to replace my rear speakers with PIONEER 4x6(120w each), but when I put the speakers in the hole, they stuck out by about 1/4". The magnet bottomed out. Which meant I could not put the panel back at the same height. I was forced to put the originals back in, and they sound terrible with new 4 ohms up front. What speakers will you be using? Let us know if they fit.
------------------
84 SE 2m4 4-sp Red/Tan 14,000 mls
87 GT V6 5-spd Ruby Red/Grey