I have the exact opposite “problem”….I can ONLY improvise.
I always played by ear only, never took the time to sit down and learn proper theory so I play only by listening and adjusting myself accordingly. It’s not that I’m playing nonsense, not at all, I use scales, phrases and different techniques but again, all by ear and “on the spot” improvisation.
However if I hit a nice lead, then I memories it and that’s what I use for the song (in my band).
I remember everything by ear, I don’t write a single note.
Now, there are 2 advice I can give you:
1. Don’t think to much, go with the flow of the song and try to “feel” the song when soloing.
2. Learn scales and phrases. It will give you all the “words” and “sentences” you need to improvise correctly.
Keep in mind that it does need practicing, play as much as you can by ear and you will see its getting better.
Good luck.