I think all the tools are trying to tell you something. Using fonts is in fact the largest dependency with many websites.

I specify only the characters I use in custom fonts. I budget no more than 30kb to fonts when making a website.

These tools complain that Google font's css is only cached for 2 hours. Which is something none of us can control. Even Google's own page speed tool complains about this. I get perfect scores in everything but Google fonts so it sticks out like an eye score.

Cant you serv them from your own server?

Google does optimisations based on user agent. Ilya Grigorik does not recommend hosting the fonts from your own server.