They jump high.
They're also pretty curious and determined.
Last night my naughtiest goat, Rosie, thought it was a fantastic idea to jump on top of her house (which is 4' plus high) and then jump over the chain-link fence enclosing the pen. She was eventually captured and brought back home where my father had dutifully moved the goat house to the middle of the pen, thereby removing any temptation to escape again.
Lesson learned: Never underestimate the jumping capabilities of a dairy goat.