Last month it was that anyone receiving research grants from the state will be banned from lobbying “government and Parliament” on either policy issues or funding. The rule is said to be aimed principally at charities, but who knows the truth: in any case it covers all government grants “related to research and development”, and that means academics like me.

You might wonder how this came about. Presumably some unsuspecting charity worker or academic frightened one struggling politician who has, in the fog of war, overreacted. The ban is hardly improved by the fact that government departments can issue specific exemptions for approved individuals: if anything, that makes the whole project more sinister. A complex, centrally administered database of academics with permission to speak to politicians will be more expensive, more unmanageable and much more bizarre than any straight blanket ban.

Let me explain my concern. I’m an academic, and in the past two weeks I’ve had meetings with two ministers, one permanent secretary, one departmental director, and various other civil servants, analysts and wonks. I can only describe the process that led up to these meetings, if not the meetings themselves, as lobbying. But I also hope this activity is in the public interest. What’s more, I know there are many academics like me, seeking out politicians and senior civil servants on everything from alcohol to poverty, forestry and more. This is part of being a public servant, but it’s a practice without a public presence, toolkit, handbook or diploma.

That’s a problem. I don’t just want this ban overturned: I want to see more academics talking to policymakers, and I want the public to know what we do, so that they can decide if it’s good or bad. That’s why I’m now going to describe, in banal human terms, the meetings I’ve had in Whitehall over the past few weeks: to pool knowledge and share techniques, to promote transparency, and to help fearful, inexperienced politicians understand that there is nothing to fear from academics in practical outdoor clothing.

����Vlog

Right now, I’m on a train to see the secretary of state for education, Nicky Morgan. Why does she care what I think? I’m a medic, and a lecturer. I work on better ways to use health data, on evidence-based policy and on the harms inflicted by bad research regulation. I’ve also sold half a million books on statistics and evidence, which probably makes me a little more visible to politicians (but, in truth, only a little). I’ve previously done work for Morgan’s department on how to get more randomised trials done in education, and that came about, in turn, because I have been writing and shouting about these topics for a decade in newspaper articles, lectures for civil servants and the public, unsolicited letters, feedback on consultations and anywhere else I think anybody will listen.

The train journey to London is boring, so I try to pack in whole days at Westminster, as I did last Tuesday. The morning was spent on the new Data and Evidence Board at the Ministry of Justice, manned by a range of academics and Civil Service analysts from many different disciplines. Next I went to see Will Cavendish, director of innovation at the Department of Health. We met to discuss how to improve reporting standards on clinical trials and better ways to get NHS data used impactfully. Did he invite me, or did I badger him? I can’t remember: either way, he was interested in the small , which has just launched a freely accessible that lets you see what every GP in the country is prescribing. I want to see more similar open source tools for the NHS.

����Vlog

Did I manipulate him, using taxpayers’ money? No. Cavendish is a grown-up who has meetings like this every day. I certainly wanted to change his views and priorities: I advocated for open, competitive funding streams for the kind of open data work we do, because I think it’s reasonably novel, useful and sits outside conventional academic funding models. I learned a little about how his corner of the department thinks.

Is that lobbying? You decide. Nobody handed anyone a giant cartoon cheque, and government policy won’t suddenly change. Will there be any concrete outputs? I can tell you what might happen. He might suggest me for a boring committee where, one month in five, you can help stop something catastrophically stupid happening. One day, in the distant future, he might phone and ask for rapid informal feedback on an idea he’s mulling over, and I’ll be happy to help. This is the banal working reality of public servants with research grants who go into Whitehall. Is this what the bill is trying to stop? Nobody seems to know.

Next, I went to spend an hour with the minister for life sciences, George Freeman. We discussed how the NHS and the state can get better insights from data, what kind of funding structures would best deliver that, what private buyer-seller markets on health data can achieve and where they deliver lower value. Whatever “lobbying” means, I was probably doing it here, in the sense that there are things I am sure would work well, in the public interest, in fields I know well. At no point did Freeman show any signs of feeling intimidated, and he’d laugh in your face if you suggested it. That’s because, like Cavendish and almost everyone else in Whitehall, he is a grown-up.

But then I might well say that, because I like him – a lot. Freeman is nerdy, and understands both business and science. Do politicians like him meet people like me so we will say nice things about them? Maybe. Being liked is part of their job, just as discussing problems in technical fields with ministers should be part of mine. But ministers also aren’t ogres, and there is a worrying trend for some academics to imagine that they are. And so, to expose myself to maximum hatred, I will now tell you about a good experience with one Cabinet Office minister that some colleagues regard as frankly satanic.

Over the past couple of years I’ve solicited various meetings with Oliver Letwin. These were all trying to address the problem of excessive regulation around low-risk clinical trials of commonly used treatments, such as statins. Why, and how? Two years ago, I made a short film for Daily Politics on BBC Two to discuss AllTrials, a policy campaign I helped set up in 2013 to address the long-standing problem of clinical trial results being routinely and legally withheld from doctors, researchers and patients. After the studio discussion, politicians from the show were happy to chat. Grant Shapps – former co-chair of the Conservative Party – made supportive noises. Chatting on the street outside Millbank studios we bumped into Letwin, who joined our conversation. This is what people mean when they talk about the Westminster village.

����Vlog

Letwin is smart, and keen on open data. After agreeing that withholding clinical trial results is absurd, he asked: “Presumably there are other equally stupid things happening in medical research right now?” And so we started discussing excessive regulation around clinical trials, with a longer subsequent conversation in his office. Since then we’ve had two big meetings he’s organised in No 9 Downing Street, with senior people from the Department of Health, the medicines regulator, the research regulators and more, to discuss reducing the regulatory burden. There will probably be minimal progress: but this is more than nothing, and it’s not something to ban.

Why does Letwin care? I suspect he views this issue as one more example of bureaucrats killing patients with red tape. I might disagree with some of his other examples (I don’t know) but on trials, I’m glad to identify the roots of a minor, short-lived, informal alliance because he’s absolutely right. Taking the onerous regulations devised for trials on novel high-risk interventions and applying them to low-risk interventions has been a thoughtless bureaucratic disaster.

What’s more, I will happily disclose that – to talk to – I like Letwin. He’s nerdy, and in meetings he’s bluff, direct and efficient. I’m glad he’s given a tiny fraction of his annual activity to a cause I hold dear. That’s not an endorsement of everything he stands for: he’s said some awful things. But while you might strongly disagree with free market ideologues on privatising the NHS, or how best to handle perverse incentives in the benefits system, they are often very keen to address things like asymmetrical information that stops markets functioning as they should. They are also the elected government.

����Vlog

And that is the kernel of pragmatic relations between academics and policymakers: to find productive common ground that serves the public good; to offer help if it’s wanted; and to be pushy if you think it’s worth it. Should this activity be banned? Of course not, and any whiff of a ban will have more timid university departments anxiously monitoring anyone who talks to a minister. But we need more of this interaction, not less, and most clueful people in Whitehall would agree. Formal government science advisers exist, and they serve many roles excellently: but they have formal civil servant positions and are limited by formal and unspoken rules on what they can do and say. What’s more, in many corners of government, policy staff are crying out to be lobbied: just this morning I’ve had one policy adviser complain bitterly that only three academics in his field have ever even sent him an email. He asked, in all seriousness: “If none of them are ever calling me, then what do they all do all day that they all think is more useful?”

Maybe you’re an academic and you think it’s not worth it. You’re absolutely right that this impact is hard to capture, and hard to cash in for the research excellence framework. If I tried to write, on my CV, all the activity I have described above in natural, honest language, I’d look like a moron, a failure or a fantasist. Maybe, in your grumpiness, you’ve read what I’ve written and think I’m just showing off. What’s more, this is all time away from meeting conventional metrics, like writing papers, which terrifies any researcher. And it costs more than time: I’ve never got round to claiming back any of the endless £29 train tickets (£67.10 before 9am) that these trips to London cost, even if I had a budget code that would match them.

But this is the real work of being a public servant. Working for free, on moderately helpful stuff that people will almost never hear about: that is the only true protest against idiotic metrics that fail to capture the true impact of academic work. So if you’re an academic who lobbies, then don’t be shy, and don’t be scared: you should share your experiences, and your techniques. If you want to waste even more time on activities with no credit and no hope of funding, then perhaps we could set up a course, or a forum, to pool knowledge on better ways to interact with Whitehall. And lastly, if you’re a politician, and you really want to ban this activity, then shame on you. You’re a failure, an obstacle to good progress, and an outlier. But there’s one final piece of happier news. You won’t last long.

Ben Goldacre is a doctor, campaigner, writer and senior clinical research fellow at the University of Oxford.

����Vlog