Nice course. Is there a way use a more complex imputer calculation, for instance to fill null value with the mean value of each agent sales instead of the mean of the whole dataset ?
Hey thanks for your comment! I don't think there's an out-the-box way of doing this in pySpark, you can achieve it though by grouping your data (or filtering out each category) and fitting the Imputer on the subset of data, before joining them back again Another method is to use the Pandas on Spark API as they have much better support for that kind of groupby + fillna functionality - read more here: spark.apache.org/docs/3.2.0/api/python/reference/pyspark.pandas/api/pyspark.pandas.groupby.GroupBy.fillna.html
![เปิดใจ “ต้นหอม ศกุนตลา” หลังโดนพาดพิงธุรกิจที่เคยทำ และผลกระทบทำถูกปลดฟ้าผ่า! | แฉ 24 ต.ค.67 [2/3]](http://i.ytimg.com/vi/vCi5OIqlxaE/mqdefault.jpg)
Nice course. Is there a way use a more complex imputer calculation, for instance to fill null value with the mean value of each agent sales instead of the mean of the whole dataset ?
Hey thanks for your comment! I don't think there's an out-the-box way of doing this in pySpark, you can achieve it though by grouping your data (or filtering out each category) and fitting the Imputer on the subset of data, before joining them back again
Another method is to use the Pandas on Spark API as they have much better support for that kind of groupby + fillna functionality - read more here: spark.apache.org/docs/3.2.0/api/python/reference/pyspark.pandas/api/pyspark.pandas.groupby.GroupBy.fillna.html