The Leeds Teaching Hospitals team, who presented at a meeting of the American College of Cardiology, described the results as "stunning".
The British Heart Foundation called for longer trials to assess the pills.
Vitamin D is vital for healthy bones and teeth and may have important health benefits throughout the body but many people are deficient.
The average age of people in the study was 70 and, like many people that age, they had low levels of vitamin D even in summer.
"They do spend less time outside, but the skin's ability to manufacture vitamin D also gets less effective [with age] and we don't really understand why that is," said consultant cardiologist Dr Klaus Witte.
Patients were given either a 100 microgram vitamin D tablet or a sugar pill placebo each day for a year.
And researchers measured the impact on heart failure - a condition in which the heart becomes too weak to pump blood properly.
The key measure was the ejection fraction, the amount of blood pumped out of the chambers of the heart with each beat.
In a healthy adult, the figure is between 60% and 70%, but only a quarter of the blood in the heart was being successfully pumped out in the heart failure patients.
But in those taking the vitamin pills, the ejection fraction increased from 26% to 34%.
Dr Witte told the BBC News website: "It's quite a big deal, that's as big as you'd expect from other more expensive treatments that we use, it's a stunning effect.
"It's as cheap as chips, has no side effects and a stunning improvement on people already on optimal medical therapy, it is the first time anyone has shown something like this in the last 15 years."
The study also showed the patients' hearts became smaller - a suggestion they are becoming more powerful and efficient.
In the UK, people over 65 are advised to take 10 microgram supplements of the vitamin.
However, Dr Witte does not think high-dose vitamin D should be routine prescribed just yet.
He told the BBC: "We're a little bit off that yet, not because I don't believe it, but data have shown improvements in heart function, they may show improvements in symptoms and we now need a large study."
It is also not clear exactly how vitamin D is improving heart function, but it is thought every cell in the body responds to the vitamin.
Most vitamin D comes from sunlight, although it is also found in oily fish, eggs and is added to some foods such as breakfast cereals.
Prof Peter Weissberg, from the British Heart Foundation, cautioned that the patients seemed no better at exercise.
And added: "A much bigger study over a longer period of time is now needed to determine whether these changes in cardiac function can translate into fewer symptoms and longer lives for heart failure patients."