This may sound pedantic, but I think we need to get some terminology straight, because I am confused about what you are saying.
The CHARGER is inside the car. So let's not use that term or anything outside the car.
The EVSE is the device that plugs into the wall (or is hardwired to the breaker panel) on one end, and has the charging cable and connector that you plug into your car on the other end. For a permanent installation this may be mounted on the wall, or it may be the the portable unit that came with your car. Tesla calls the wall mounted version of this a WALL CONNECTOR and the portable one that comes with the car a UMC (Universal Mobile Connector). The Tesla does NOT come with a WALL CONNECTOR (this a a $500 item, which is actually a pretty good deal - https://shop.tesla.com/product/wall-connector), but it does come with a UMC.
The J1772 ADAPTER is a small device that has a female J1772 connector on one end and a male Tesla connector on the other. You would attach this to the end of a standard EVSE connector so that you can plug it into your Tesla. This adapter comes with the car. If you want a spare, it costs $95 from Tesla (https://shop.tesla.com/product/sae-j177 ... ng-adapter) although I found one online for a bit less.
The NEMA 14-50 ADAPTER is a small cord with a NEMA 14-50 plug on it that plugs into the Tesla UMC so that you can plug it into a NEMA 14-50 (240V) outlet. There are other similar adapters that can be used to plug into other 120V and 240V outlets. They used to include the NEMA 14-50 ADAPTER with the vehicle, but that is now something you need to buy extra (about $35 - https://shop.tesla.com/product/gen-2-nema-adapters)
With all that out of the way, here is where you lost me: "Another issue is I currently have my connector hooked up to wall charger." What are you calling a connector and what are you calling a wall charger? I'm assuming you just mean that you have a wall mounted EVSE that you insert the connector into and it holds it neatly in place. It's not actually "connected" in an electrical sense. It's just a plastic piece that neatly holds the connector in place. Let's call this the HOLSTER, although in your case it's built into to your EVSE. You can get these standalone for $15 or so (https://www.amazon.com/J1772-electric-h ... B00TBW1NRM). You can get similar for the Tesla connector from Amazon (https://www.amazon.com/Tesla-Motors-102 ... 01EO5BCR6/) but most people just 3D print their own, or like me, just lay the cord over the charging station. It doesn't NEED to hook to anything. It is true though that with the J1772 ADAPTER attached, it will not fit into your EVSE's holster. You will have to remove it each time, or like I do, just lay the cord over the EVSE.