Pulsars have long been predicted to be sources of electron/positron (e-e+) pairs. Highly energetic e-e+ in the wind regions, or inner magnetospheres of pulsars, and also when released into the interstellar medium, can inverse-Compton (IC) scatter ambient photons to very high energies. Recent HAWC observations of extended TeV-scale gamma-ray emission around Geminga and Monogem have provided indirect evidence for TeV-scale e-e+ acceleration in young pulsar nebulae. However, predicting and observing high energy gamma-ray emission from rotation-powered millisecond pulsars (MSPs) presents a number of challenges compared to that of the non-recycled pulsar population. During this talk, I will summarize these challenges, and present our recent exciting results where we find evidence for very high energy gamma ray emission from populations of MSPs residing in globular clusters of the Milky Way. I will further highlight the unique capabilities of the Cherenkov Telescope Array to detect the expected IC signal from a putative population of 10 to 50 thousand MSPs in the center of our Galaxy.