Monitoring of Spark emissions via Spark plugin

Hi,

I am working on a prototype of a Spark plugin to report the energy consumption of executors. The logic behind is similar to CodeCarbon's, although the computation method differs slightly: the executor process scheduling is sampled regularly, converted to Wh with the TDP (provided or inferred), and aggregated by the driver. The total energy is published as a Spark metric, accessible via the REST API.

I wanted to know if you'd be interested in integrating it with CodeCarbon, for example with Spark cluster as a new type of resource alongside CPU, GPU, or RAM. It would let CC factor in the energy mix and cloud provider data, which could be cumbersome to access from a private Spark cluster (it's better not to assume internet connectivity). And it would benefit from CC's ease of use, which is a strong factor of adoption.

In any case, it's a prototype, it needs more testing and validation, and only handles CPU for now (but many data engineering pipelines don't use GPUs anyway). Here it is: https://github.com/tvial/ccspark (Apache 2.0 license). Note that it embeds your CPU database for the TDPs, I'm open to remove it if you think it's a bad idea :)

Let me know if it can be of any help Thanks