It’s a tried and tested technique used by writers and poets, but can psychology explain why first moments after waking can be among our most imaginative?

It is 6.06am and I’m typing this in my pyjamas. I awoke at 6.04am, walked from the bedroom to the study, switched on my computer and got to work immediately. This is unusual behaviour for me. However, it’s a tried and tested technique for enhancing creativity, long used by writers, poets and others, including the inventor Benjamin Franklin. And psychology research appears to back this up, providing an explanation for why we might be at our most creative when our minds are still emerging from the realm of sleep.

The best evidence we have of our mental state when we're asleep is that strange phenomenon called dreaming. Much remains unknown about dreams, but one thing that is certain is that they are weird. Also listening to other people's dreams can be deadly boring. They go on and on about how they were on a train, but it wasn't a train, it was a dinner party, and their brother was there, as well as a girl they haven't spoken to since they were nine, and... yawn. To the dreamer this all seems very important and somehow connected. To the rest of us it sounds like nonsense, and tedious nonsense at that.

Yet these bizarre monologues do highlight an interesting aspect of the dream world: the creation of connections between things that didn't seem connected before. When you think about it, this isn't too unlike a description of what creative people do in their work – connecting ideas and concepts that nobody thought to connect before in a way that appears to make sense.

No wonder some people value the immediate, post-sleep, dreamlike mental state – known as sleep inertia or the hypnopompic state – so highly. It allows them to infuse their waking, directed thoughts with a dusting of dreamworld magic. Later in the day, waking consciousness assumes complete control, which is a good thing as it allows us to go about our day evaluating situations, making plans, pursuing goals and dealing rationally with the world. Life would be challenging indeed if we were constantly hallucinating, believing the impossible or losing sense of what we were doing like we do when we're dreaming. But perhaps the rational grip of daytime consciousness can at times be too strong, especially if your work could benefit from the feckless, distractible, inconsistent, manic, but sometimes inspired nature of its rebellious sleepy twin.

Scientific methods – by necessity methodical and precise – might not seem the best of tools for investigating sleep consciousness. Yet in 2007 Matthew Walker, now of the University of California at Berkeley, and colleagues carried out a study that helps illustrate the power of sleep to foster unusual connections, or “remote associates” as psychologists call them.

Under the inference

Subjects were presented with pairs of six abstract patterns A, B, C, D, E and F. Through trial and error they were taught the basics of a hierarchy, which dictated they should select A over B, B over C, C over D, D over E, and E over F. The researchers called these the “premise pairs”. While participants learnt these during their training period, they were not explicitly taught that because A was better than B, and B better than C, that they should infer A to be better than C, for example. This hidden order implied relationships, described by Walker as “inference pairs”, were designed to mimic the remote associates that drive creativity.

Participants who were tested 20 minutes after training got 90% of premise pairs but only around 50% of inference pairs right – the same fraction you or I would get if we went into the task without any training and just guessed.