Here are my ideas on how we are doing this in Weaviate - https://weaviate.io/blog/2022/11/ref2vec-centroid.html. Really enjoyed this, thanks for the dataset recommendation! Would love to continue chatting about this in the Weaviate slack channel if that interests you!