"I've never discussed it with anybody before - intentionally," the 66-year-old singer-songwriter told The Associated Press on Monday during a break from recording. "I thought maybe I would tell it to Caroline when I met her someday."
He got his chance last week when he performed the song via satellite at Caroline Kennedy Schlossberg's 50th birthday party.
Diamond was a "young, broke songwriter" when a photo of the president's daughter in a news magazine caught his eye.
"It was a picture of a little girl dressed to the nines in her riding gear, next to her pony," Diamond recalled. "It was such an innocent, wonderful picture, I immediately felt there was a song in there."
Years later, holed up in a hotel in Memphis, he would write the words and music in less an hour.