Liberal senator James Paterson said he was “concerned” that activist Drew Pavlou was escorted from Parliament House by federal police yesterday after the pair had met, but the AFP and the parliament won’t say why he was asked to leave the building.

Police officers told Pavlou he was considered an “active protester”, while sitting in a public cafe in Parliament House drinking a coffee, and that “higher ups” had said they wanted him to leave the building.

“An Australian federal police team interrupted me while I was quietly having lunch at Parliament House and directed me to leave parliament or face trespass charges as a ‘high risk individual’,” Pavlou said.

The Brisbane man is an anti-Communist party activist who has staged high-profile protests against Chinese human rights abuses and for pro-democracy causes in Hong Kong, including interrupting July’s Wimbledon tennis final by holding a sign and wearing a shirt reading “where is Peng Shuai?”.

He was arrested in London later that month after staging a small protest outside the Chinese embassy, holding a Uyghur flag. Pavlou said he was arrested over a false “bomb threat” delivered to the Chinese embassy that he claims came from a fake email address designed to frame him.

On Wednesday, Pavlou was in Canberra to meet Paterson, who is the shadow minister for countering foreign interference and a former chair of parliament’s intelligence committee, as well as the committee’s current chair, Labor MP Peter Khalil. He had also spoken to activists on the front lawn of Parliament House who were protesting in support of the Uyghur minority.

Pavlou said he had moved to the Queen’s Terrace cafe, in a public area of Parliament House, after his meeting with Paterson and while he waited to meet Khalil. In tweets published about 1.30pm, Pavlou claimed police had asked him to leave the building 

In video of the incident, seen by Guardian Australia, two AFP officers approached Pavlou and asked him to leave Parliament House until his meeting later in the day.

“Come back at six o’clock, that’s fine,” one said.

When Pavlou asked why he was being asked to leave, another officer responded “because you’re an active protester, they don’t allow active protesters in the house.” Pavlou responded that he wasn’t protesting, just eating lunch.

“You’ve mentioned what your intentions are for the rest of the day,” an officer replied.

“That’s just the policy and thats come down from the boss. We’re happy for you to come for your meeting but hanging out here outside of that …”