There are several possible causes of vertigo. The one that I am most familiar with is caused by loose particles in the ear canal, that when moved, bounce off the small hairs in the ear canal that are responsible for positional awareness (keeping your balance). These loose particles will sit stationary for a while and then, when disturbed, will bounce off those sensors and wreak havoc with your balance. Over time, the loose particles will be absorbed but until then, one will experience vertigo without warning. This kind of vertigo can be diagnosed by looking in a mirror when you are subject to an attach and see if one eye is moving back and forth (slightly). This is the eye trying to make sense of the information that the ear is giving it. The eye that is moving will correspond to the ear that is causing the problem. There are positional exercises that one can do to move the loose particles to a corner of the ear canal that is not likely to cause problems. They can be found on the web if one looks.
I have not had this kind of vertigo myself but have had several friends that have experienced it. It is reasonably common with people who fly aerobatics but I have known people who have had it who had not been doing anything that would account for it.
But, there are other types of vertigo so this may not apply to you.