Ever since I started teaching college, about 13 years ago, people continually ask what in the world I do all summer when the students are gone.
The list is nearly endless. Teaching-related usually comes stuff first. My courses from the year need to be "put to bed," which involves making notes for my future self on what worked and what didn't, new ideas I had for next time, and lots of sorting and filing. I also plan for next year's courses by reviewing those handy notes from my past self, new ideas I'd like to steal from colleagues and the web, etc. I also catch up on e-mail (e.g., notes from alumni, administrative tasks), write reference letters, update my web pages, and read articles that I've put off all year. For my dean-related stuff, I still attend planning meetings, review student evaluations for faculty members, analyze data regarding teaching and learning, develop an annual calendar (when various reports and contract recommendations are due, etc.), plan workshops, recruit mentors for new faculty members, review course syllabi, revise orientation materials, review faculty members' annual plans, etc.
The beginning of summer is often hard because the main source of energy is gone: the students. By the end of summer, though, I get used to the quiet and it's hard to adjust to having students come back with all their noise and excitement. I suppose summer is like fasting, where the routine changes and eventually you get used to feeling hungry; when you start eating again it's hard to digest the food at first, but you quickly adjust and hopefully put into place what you've learned during the fast.