The incident -- reportedly related to involvement with prostitutes in Cartagena -- overshadowed the start of the sixth Summit of the Americas, where the president was to focus on trade, energy and regional security.
Before the president's arrival, an undisclosed number of Secret Service agents were relieved of duty and replaced, said Edwin Donovan, an agency spokesman.
"There have been allegations of misconduct made against the Secret Service in Cartagena, Colombia, prior to the president's trip," Donovan said in a statement.
"Because of this, those personnel are being relieved of their assignments, returned to their place of duty, and are being replaced by other Secret Service personnel. The Secret Service takes all allegations of misconduct seriously."
The allegations involved prostitution, according to Ronald Kessler, a former Washington Post reporter and author of "In the President's Secret Service: Behind the Scenes With Agents in the Line of Fire and the Presidents They Protect."
"One of the agents did not pay one of the prostitutes, and she complained to the police," Kessler said.
Calling it "clearly the biggest scandal in Secret Service history," Kessler said 12 agents are accused of involvement in the incident "in one degree or another," from allegedly interfering in the investigation to participating in other alleged misconduct.
Kessler did not identify to CNN who provided him with details of the investigation, and CNN could not immediately confirm the claim.
The Washington Post, which was the first to report the story, said it was alerted to the investigation by Kessler.
Donovan declined to identify the nature of the alleged misconduct, saying only that the matter was being turned over to the agency's internal affairs department.
The Washington Post reported that Jon Adler, president of the Federal Law Enforcement Officers Association, said the accusations relate to at least one agent having involvement with prostitutes in Cartagena.
Adler later issued a statement saying he was disheartened by what he called "Kessler's need to orchestrate an indictment by rumor." It was not clear whether he was disputing the newspaper's characterization of his comments.
"I respect our due process and trust the Secret Service to investigate this matter professionally. I stand by the brave men and women of the Secret Service, and ask that everyone reserve judgment until the matter is properly reviewed," Adler's statement said. "It would be both reckless and premature to jump to judgment that either the president's safety or his mission in Colombia were jeopardized by the allegations in question."
A spokesman for Colombia's National Police declined to comment, referring questions to the Secret Service.
The president arrived in the Colombian coastal resort city Friday, a visit that will mark the most time a U.S. president has spent in that country, where security concerns had limited previous presidential trips.
Amid the reports that Secret Service agents were being replaced, two small blasts occurred nearly back-to-back in Cartagena.
The explosions, one near a bus station and another near a shopping mall, occurred well away from where the world leaders were gathering for the start of the summit, said Alberto Cantihho Toncell, a spokesman for the Colombia National Police.
There were no casualties, and only minor damage was reported, Toncell said.
The explosions came on the heels of a similar one earlier in the day near the U.S. Embassy in the capital city of Bogota, authorities said.
The blasts were a reminder of the violence that has gripped Colombia as it battled powerful cocaine drug cartels. Violence has significantly fallen off in recent years as the Bogota government, aided by U.S. extradition efforts, has successfully picked apart the cartels.
More than 7,600 police officers and thousands more troops have been deployed in the walled colonial city of Cartagena as part of stepped up security for the summit.
Submarines are patrolling in the coastal waters near the city, armed helicopters are hovering at the ready and snipers in strategic locations are watching for suspicious activity, officials said before the summit's start. Anti-explosive robots and radiation detectors are also part of the security detail.
(CNN's Randi Kaye, Chelsea J. Carter, Mike Ahlers and journalists Jorge Baron and Fernando Ramos contributed to this report.)