While it may or may not meet your contextual definition of ‘rock’,… lead tastes somewhat sweet, apparently.

The Romans boiled grape juice in lead pots to produce a kind of syrup that was used to sweeten wine.

Lead is uh, a neurotoxin and likely carcinogen, so probably don’t lick the sweet rocks too much.

According to:

galleries.com/minerals/property/taste.htm

… apparently borax tastes sweet and… alkaline?

Chalcantite is described as ‘sweet metalic and slightly poisonous.’

Melanterite is apparently ‘sweet, astringent and metallic.’