First of all, great idea for a thread, I hope we get some more input, I'm always curious about how others create.
What follows are certainly not hard fast rule. These are just the mechanisms I've come to use to streamline my own creative process after years of trial and error. YMMV.
Writing music or words is all about asking the right questions. Ask the right question, then restate the question in the answer.
Think in the most explicit terms possible. State your purpose. Refine the statement. Test and retry.
I'll try and flesh this out with hypotheticals to better explain what I mean.
What emotion are you trying to convey? Hurt, anger, longing, loss, rejection, regret etc.....ok...what does that emotion "sound" like....
Lets say in this first example it is a feeling that inspires the process. You choose loss, you're writing something about losing something...a person, a pet, a girl, a car, a guitar whatever....
What's your universal statement? i.e. I've experienced loss and it hurts, but pain can make us stronger and help us help others
how do you make this resonate with someone else? How do you take your universal concept and make it resonate on the individual level i.e. loss effects me specifically in this way:
"When ____ was taken from me, it made me feel like I wasn't alive, like my skin was wrapped around something dead, like my heart and soul had been torn apart, I never wanted to have to feel anything again"
Now make it more precise. Condense it. Make as explicit as possible. Always say the most with the least.
"When ____ was taken from me, I was dead inside my calloused skin, an empty heart torn apart, I never wanted to feel again"
How does that sound? What chords, rhythms, melodies or sounds help express that. What soundscape takes you to that place.
Now give that statement life. Make this real to the listener. What did it sound like, smell like, look like.... what was it like being alive wanting to be numb? How did it effect your day to day life, your mindset, your relationships with other people?
I felt like no one could reach me, utterly alone, I just couldn't believe that I'd lost the chance to experience life with _________.
Condense.
"Unreachable in broken state, unthinkable that twisted fate, could blanket cold and lasting calm the memories that I'd counted on."
What's the next question here? The right question? For me in this instance, the next thing I'd want to know from the writer is how did you deal with that?
From here, you just keep asking the right question, build a chorus around your universal principle. The verse is where you relate it on an individual level. Bridges tend to be good places to wax philosophic.
If you have writers that you like, look up their songs and try to determine the implied questions that lead you from one line to the next because whether or not they're aware of it (some people just have really natural flow) they are asking themselves questions. You'll begin to see a pattern of thought, of how they work through a subject. A good story teller is seldom interrupted by questions because they will answer them before they can be asked. Which brings me to this: sometimes the right question is what does this make you think of next, but sometimes, imagine youre the listener...if you knew nothing about the story you were telling...what would you want to know? What details might make this easier to relate to or help with imagery.....or maybe In what order would you want to know it?
Music can be done the exact same way.
You have a riff, you want to write the next riff, just start throwing out words until you arrive at a phrase, then uncover the statement statement....
This riff sounds like violence.....like a fight....but it's personal....this riff sounds like fighting someone you've known.....someone you have history with....someone who has slighted you....and its fresh in your mind.....you really don't like this person. Condense it.
This riff sounds like you're finally beating the shit out of someone you hate. That's a pretty good universal statement. Everyone can get a pull on that. So how did you come to meet this person, how'd they cross you....what does that sound like....fast and aggressive, open, empty, was it tense in the beginning? The end? Did it start out good...etc etc. Then create the soundscape. A lot of the time if you can just be open to it, in addition to helping you with the next riff they can also do a lot of the actual writing for you.
Or maybe this riff sounds like missing someone.....but kind melancholy....like smiling because of the pain.....like maybe there's a shade regret....there's hurt...but its not all pain....it sounds like it hurts because you once cared about this person a great deal.
Condense.
This riff sounds like regretting how you treated an a past love.
Is that a universal statement, or an individual revelation? Both? Quick sell that shit to kanye and cash in!
Now start asking the right questions.
How'd it start, how good were the good times, how bad were the bad times....did you fuck up? Did she/he? How so? Were the consequences permanent? Did you learn anything from this experience? Do you ever dream of a second chance? Did it change the way you treated people in the future? Did it change you both or just you or just him/her? Is that why it would never work now?
What do those things sound like to you? What chords, melodies and sounds make those emotions real?
Writing both words and music is intensely personal and I know not everything works for every body but I hope this helps.
Good luck!