Toronto Mayor Rob Ford’s chief of staff Mark Towhey was fired Thursday and escorted out of city hall by security guards.

"I did not resign,” Towhey told reporters as he left the building.

Ford’s office confirmed Towhey was no longer working as chief of staff in a statement released Thursday afternoon.

“Mr. Towhey has been an intricate part of the Mayor's Office and has made many valuable contributions,” the brief statement said. “The Mayor thanks Mr. Towhey for his valuable service and wishes him the very best in his future endeavours.”

Towhey was promoted to the job last summer. He was the third chief of staff since Ford became mayor.

Earl Provost, deputy chief of staff, will assume the role of acting chief of staff until further notice.

Towhey’s exit comes as calls continue for Ford to address week-old allegations about a video in which he is reported to be shown smoking from a glass pipe.

The mayor has remained largely silent on the accusations, which he called “ridiculous” late last week.

Deputy Mayor Doug Holyday said Towhey’s departure is “not going to help” the ongoing controversy.

“But on the other hand, I suggested to the mayor that he needs to get out in front and speak to the media on this matter, and he’s not done it yet,” Holyday told CTV News Channel.

Toronto Coun. Josh Matlow told CP24 that the allegations against the mayor, which have made international headlines, have brought negative attention to the city.

“It doesn’t do a good service for Toronto. I hope that the mayor will come forward and just address this. Help us move forward,” Matlow said.

“We can’t force somebody to speak. That’s the mayor’s decision whether or not he’d like to. But it’s not good enough just to get your brother to go and do a press conference.”

On Wednesday, Coun. Doug Ford addressed the allegations for the first time, telling reporters that the mayor told him the allegations were “untrue.”

“Rob’s telling me these stories are untrue, that these accusations are ridiculous,” Doug Ford said. “And I believe him.”

He added that his brother has been subject to intense media scrutiny ever since he took office and that the mayor would not be “pressured” into making a statement regarding the alleged video.

When asked if Doug Ford’s statement on behalf of his brother is good enough, Holyday said, “No, I don’t think so.

“I think Rob has got to settle this himself. If he doesn’t get out and state something then the Toronto Star story is what’s left for people to deal with. If he wants to contradict that and put out his facts on the matter, then people can judge. But until he does that, they’re not going to have that opportunity.”

Holyday said he still has confidence in the Mayor because “substantiated proof” about the mayor’s alleged drug use has not come out.

“People have seen a video on a small phone screen in the back seat of a car controlled by drug dealers. There’s a lot of room for error on this,” he said. “But on the other hand, the mayor’s handling of this is not exactly what I think it should be.”

Toronto Coun. Ana Bailao, who was charged with impaired driving last year and after several months publicly took responsibility for her actions, also called on the mayor Thursday to address the allegations, saying that the scandal has been “distracting” for councillors.

“I think Torontonians want to hear from the mayor,” she said. “These are very, very serious allegations.

“It’s very distracting for city hall… Us councillors are making an effort to concentrate on our business, on our work. But it’s obviously distracting to have this big cloud on top of our heads and not being addressed.”

Coun. Paul Ainslie on Thursday told reporters that he blew a warning on a recent R.I.D.E. check and lost his license for three days. He said he went public with his story – he was not charged – to refute allegations on Twitter that he had been “popped with a DUI.”

However, he also had a message for the mayor.

“My message to Rob Ford is, I don't know anything about the video as no one else has seen the video,” Ainslie said. “But I think he needs to make a statement one way or another and clear up the matter at hand.”

Reports of the alleged video first surfaced last week on U.S. website Gawker and later in the Star.

Both media outlets said they had viewed a video that appears to show the mayor smoking crack cocaine. The Star reporters also allege that in the video, the mayor makes offensive remarks about Liberal Leader Justin Trudeau and members of the Don Bosco Eagles -- a local high school football team he had coached for years.

Ford was relieved of his coaching duties on Wednesday by the Toronto Catholic District School Board, which said it would “pursue a different direction” with a new volunteer coach.

A board spokesperson said the decision had nothing to do with the current controversy around the alleged video.

TCDSB had been reviewing the mayor’s coaching role at Don Bosco since he made disparaging remarks about the school and its football players during a media interview in March, suggesting it was a “tough school” where a lot of kids “come from gangs” and “broken homes.”

The existence and content of the video has not been authenticated by CTV News.

With files from CTV Toronto's Natalie Johnson and CP24