Lee Randall: Out of uniform

Right about the time I ask Brian Paddick what song he'd choose to sum up his life, I realise I've done something hideously wrong with my tape recorder and much of our long conversation is gone with the wind. Silver linings being what they are, this affords him an opportunity to demonstrate his capacity for empathy – a trait he puts near the top of his personality CV. It helped make him a successful cop and will, he says, also make him an excellent mayor of London.

He consults with his diary planners and returns bearing tea and the gift of another hour – his lunch hour; but hunger's less important than a damsel in distress. Never mind that it's the second day of his mayoral campaign under the Liberal-Democrat banner, or that his schedule includes promotional duties for his memoir, In the Line of Fire, as well as training for April's London marathon. He brushes away my thanks, saying, "If it wasn't inappropriate, I'd hug you right now."

What a telling remark. Given the frequency, during his 30-year career, with which he was called "upstairs" for a dressing-down after speaking out about controversial police matters, or found himself the subject of screeching headlines, you'd be forgiven for thinking he's a maverick. But despite being philosophically inquisitive enough to have won the soubriquet Descartes of Dock Green, Paddick assures me he plays by the rules to an extreme degree. If the sign says keep off the grass, he'll stick to the pavement. "I've never taken drugs because they're illegal," he says. This begs the silent question: what if they weren't? And the question I voice: "What if you'd come of age before homosexuality was legalised (in 1967 in England]? Would you have ignored your sexuality and remained celibate?" There's no answer, just what I take to be a conflicted look in his eyes.

Is this good behaviour a bargain with the universe? If I toe the line on all these points, you allow me my sexuality? He dismisses my theory, saying he thrives on regimentation and routine. Without it, he speculates, "I feel I could maybe go off the rails."

How? I try picturing this composed, immaculately dressed man, who insists he values honesty and integrity above all else, doing a Pete Doherty, but it's no use. He likes shopping, but would never exceed his means. He doesn't get drunk, and as for drugs, he doesn't comment further, but drugs would mean relinquishing control, and he's fond of staying in control.

Personally, I don't think a quick hug at this juncture would tarnish us, but his rulebook bears a footnote governing conduct with an interlocutor of the opposite sex, part of a subsection on touching, generally. He is, he assures me, very tactile with his partner of nearly two years, but alert to what's appropriate with others. (This must be some rulebook. When I go to shake his hand goodbye he looks appalled.)

Technical difficulties surmounted, Paddick chooses as his defining song Shirley Bassey's rendition of I Am What I Am. Does he know it's from the gay musical La Cage aux Folles? "Is it?" He raises an eyebrow.

So who is he? The younger of non-identical twin boys born in 1958 in Balham, he was an unhappy kid, desperate for affection. His parents, Anthony and Evelyn, wed during the Second World War and spent their first four years apart while his father fought with the Eighth Army. By 1958 they had an 11-year-old son, Graham, and John and Brian's arrival was unexpected. Politically they were arch Conservatives; personally, they were undemonstrative in word and deed.

Vying for attention, Paddick assumed the role of bad twin. When I ask what kind of shenanigans he specialised in, he says only that he and John acted up whenever their parents threw dinner parties, angry that strangers were getting the attention they craved but were regularly denied.

School was a slog and he was bullied. When I ask if he was an effeminate little boy he says not particularly, but he was always neat as a pin and a bit of a swot – not qualities likely to endear you to the playground set.

Even before he understood what sex was, he knew he was gay. Aged ten, on a Wolf Cub overnighter, he caught sight of a naked older boy. "I thought, 'That's gorgeous,'" he writes in his memoir. He kept his own counsel on this, which was sensible, since his mum once announced, "I could forgive my son for getting a girl pregnant, but I would never forgive him if he were homosexual." Aged 21, he finally lost his virginity to a fellow trainee at police academy, and more time passed before he slept with a woman.

After his five-year marriage collapsed, he outed himself to his twin. "Whatever you do, don't tell mum and dad," was his brother's response, advice he ignored. Anthony took it so badly that Evelyn forbade Paddick to contact him for several weeks. His dad has since died and even now, Paddick says, the family is not entirely at ease with his sexuality.

He joined the police because his grades weren't good enough for medical school. He reckoned it was a job for life, with a good pension, and he was fed up with study. Ironic, then, that he spent roughly seven of his first 14 years engaged in higher education, earning a 2.1 in Politics, Philosophy and Economics at Oxford, and an MBA from Warwick Business School.

While acknowledging the amazing opportunities he'd never have enjoyed were it not for the police, and admitting, too, that he doesn't need to work anymore, he reserves the right to criticise. The force, he writes, is still predominantly male-dominated, macho and white. "The restrictions placed on the freedom of even the most senior officers to comment openly and publicly about the Metropolitan Police have become more draconian – anything which is seen to question or criticise government is swiftly and unequivocally corrected."

Despite excelling in one high-profile, high-responsibility role after another, he says he was "monstered". "I was portrayed not only as some kind of dangerous destabilising demon, but also as the sole cause of increases in crime across London."

And that was just the media. Within the force he endured generalised jealous backstabbing plus a special helping of homophobic bullying. But isn't it the case that 17 police forces made Stonewall's Top 100 Employers list this year? He flashes a pained expression before pointing out that companies apply to take the survey, then self-report. (When I ring Stonewall, they say the top 20 are then independently audited, and that next year's selection process will be revamped.)

Paddick says, "There are two things in any big organisation, one is changing policy and leadership at the top and the other is changing how it feels to people. There's always a time lag between the two and in society, generally. The thing that gets me about some senior police officers is that they don't really mean it.

"There's not enough effort being put into dealing with homophobia. There isn't the will to deal with it at the senior level. Too many people are saying that doesn't matter, we've just got to get on with the job. But it is important and it does matter. The system in Britain is policing by consent, and if you don't maintain the trust and confidence of the communities, particularly in London, then the police will become ineffective. Dealing with racism, homophobia and sexism is absolutely crucial to police effectiveness."

He does this sometimes, slips into candidate mode, which is understandable. Whether prompted by hubris or superstition, Paddick has cleared his diary from 2 May onwards, saying, "I'm planning on becoming mayor and I thought I could be very busy." (He certainly would be, given his pledge to cut crime by 20 per cent or stand down after one term.)

It's Paddick's self-confidence – not to be confused with arrogance – that helped propel him up the ranks at speed, and gave him the courage to make controversial decisions, such as the choice, while Commander of Lambeth, not to arrest those carrying small amounts of cannabis, a "softly, softly" approach that, for all the flak it engendered, saved thousands of man hours and increased arrests for Class A drugs.

"People ask, how can you be so confident, as a leader, to make these really radical decisions? The answer is that I'm confident that if I've got it wrong people will tell me, or even if they don't tell me, I'll pick it up. A lot of leaders are very nervous and want to make sure that every decision is absolutely right and that they can't possibly be criticised. As a consequence they rarely make a decision or it's a minute change from the status quo. I'm not afraid of making mistakes." Or getting a reputation for indecisiveness? "No, because I'm not stupid and most of the time I get it right." With that he laughs long and hard.

As a young man Paddick went through a religious phase. Does he still keep the faith? With a big sigh, he pulls back in his chair. "I can't unbelieve things because they're inconvenient. There's a lot in the Christian religion that wags a finger at the gay lifestyle, but I can't help being gay and I can't help believing in God and Jesus Christ, so there's a conflict. I still believe but I don't go to church apart from weddings and funerals."

Did he go gung-ho for God because he craved a sense of belonging? "Not consciously, and I had a very negative stereotype of Christians. I'd had enough problems being bullied at school without being Christian as well, as it were. It wasn't an easy decision to make or something that I was lusting after. A bit like being gay – life would be a lot less complicated if I didn't believe."

Is he saying if he could change things he wouldn't be gay? More thoughtful silence, then: "Um, life is more complicated if you're gay. Having said that, you become more self-aware and understand other people better. There was a black preacher who gave evidence for the Lawrence inquiry, who said most black people feel over-policed and under-protected, and I thought that's exactly how I feel as a gay man, even though I'm a police officer. So I think it helps you to empathise with people from minority backgrounds.

"I've had public rows – very genteel – with Matthew Parris, for example, who says that gays have never had it so good. Well, in the social circles that he moves in, maybe, the liberal intelligentsia bubble in arguably the most cosmopolitan city in the world. But in the last few years people have been murdered because they're gay, so it's not straightforward."

It's one thing lecturing about equality and empathy, better still to be a role model. "You can say, 'Believe it or not, I'm gay and I do a good job.' But the example is more powerful than words, so if there are ways in which I can be a positive example to try and change perceptions of gay people, that's far more powerful than one of the other candidates saying they believe in equal rights for gays."

All the more reason, I say, why it's vital for high-achieving gays and lesbians to come out. "That's what David and Elton think," he replies, "that it's very important to mainstream gay people – we're not all hairdressers and air stewards. There are successful gay people in all walks of life, not because they're gay, just because of the kind of people they are."

That would be David Furnish and Elton John, friends since their introduction by one of Paddick's former long-term boyfriends. Through them he's befriended a raft of celebrities, including George Michael, who routinely greets him with the salutation, "Got a spliff?" This is not so much a reference to Michael's passion for weed as it is a wry nod to the kiss-and-tell story Paddick's ex, James Renolleau, sold the Mail on Sunday for £100,000. In it he accused Paddick of sexual indiscretions, and claimed he smoked pot on numerous occasions. Paddick went into battle mode and now jokes that he and Elton John are members of a very exclusive club – people who've successfully sued Associated Newspapers.

So it's odd, if not downright cynical, that first serial rights for this book went to that newspaper. He doesn't seem to absorb this, just talks about being wined and dined and joking to the paper's proprietor that there'd be no need to extract juicy relationship stuff from the manuscript, "since they'd printed that already".

What is most striking about his relationship with Renolleau is how toxic it became. As he lay on the ground outside their flat being punched and kicked by his lover, Paddick prayed nobody would call the cops – he was one, after all. It's hard enough for people to understand domestic violence, harder still to grasp how it happened to him. He nods. "It's a very subtle process. I'm 6ft 2in, 13 stone, educated and so forth. But at the same time I have an emotional vulnerability, which is what I think enables me to empathise with other people. It makes great leaders; it's where conviction politics comes from, where you get emotional about issues. But you can't turn your vulnerability on and off, and if you get into a relationship with someone who is prepared to ruthlessly exploit it, that's what's going to happen."

Surely people asked him the foolish question posed to so many women: why didn't you leave? "I used to deal with domestic violence regularly and could never understand why, by the time the stitches had been put in by the casualty doctor, the victim was in love with her partner again and wanted to forget all about what happened. Until – obviously not as extreme – I was in a similar position. And then it was a case of, 'Oh now I get it,' though only after I'd come out of it. While I was in it I didn't even recognise what was going on."

These days he wears a gold band on his right-hand ring finger, and seems all loved up. Not many details are forthcoming about Petter. I know he's Norwegian and I'm guessing, from the number of times Paddick refers to him as "my long-suffering partner" and "my poor partner" that patience is one of his virtues. For instance, when Paddick turns 50 on 24 April, he's scheduled to take part in Question Time, and still hasn't decided if it's better to negotiate with Petter about attending the after-show dinner, or the BBC about providing Champagne and cake.

Long after we part I'm still contemplating Paddick. This is no bad thing. Many people are all surface and no depth, though having said that, his surface is ridiculously, distractingly attractive. Without the greying hair he'd easily pass for 30, and his physique is sleek as a greyhound's. He is warm and engaging one minute, slumped in his chair or laughing heartily. Then suddenly he's prim and proper again. Perhaps that comes from years as a commander, walking the fine but vital line between accessibility and over-familiarity. I'd like to get him giddy and hear his secret thoughts, but I suspect he'd cite a rule forbidding it.

In the Line of Fire is published by Simon & Schuster, priced £17.99.