c***z
发帖数: 6348 | 1 就是我们公司有一个model，预测网上shopping cart里面会有什么东西
我改进的model比老model RMSE要低
但是老model把cart size normalize to 1，虽然没什么依据
于是让我把新model也normalize，结果RMSE显示老model好
虽然我解释说因为normalization distorted data at local level所以RMSE不再是有
效的指示，并且用了US coast line 和 加州 coast line谁更长做例子，但是自己感觉
不是很hit the point
大家有什么idea么？谢谢啦 |
A*******s
发帖数: 3942 | 2 没看明白
“shopping cart里面会有什么东西”这个算怎样的random variable？multiclass？
如果是multiclass的又怎么用RMSE来衡量？
【在 c***z 的大作中提到】
: 就是我们公司有一个model，预测网上shopping cart里面会有什么东西
: 我改进的model比老model RMSE要低
: 但是老model把cart size normalize to 1，虽然没什么依据
: 于是让我把新model也normalize，结果RMSE显示老model好
: 虽然我解释说因为normalization distorted data at local level所以RMSE不再是有
: 效的指示，并且用了US coast line 和 加州 coast line谁更长做例子，但是自己感觉
: 不是很hit the point
: 大家有什么idea么？谢谢啦
|
c***z
发帖数: 6348 | 3 My model (a decision tree) outperforms the old model if the unit of analysis
is items bought
(it should, since the old model predicts that everything the person viewed
is bought)
the old model outperforms when we fix the cart size to be 1 (the old model
then predicts that 1/n of each item viewed is bought, where n = number of
items viewed)
I am not comfortable about fixing the cart size to be 1 at the first place... |
r*****d
发帖数: 346 | 4 "预测网上shopping cart里面会有什么东西"还不够精确，要看你们究竟对什么感兴趣
，才能建立loss function比较model.
举个例子，chaoz的cart里有两个kindle一个电子琴。如果我的预测是with 100%
probability chaoz的cart里有一个kindle一个电子琴，那么loss function应该扣分吗
？如果不扣分那就说明你们只对yes/no感兴趣对数量并不感兴趣，任何不感兴趣的信息
最好不要出现在prediction里，因为presumably能简化model而又抓住了重点。
继续chaoz的cart里有两个kindle一个电子琴的例子，你们会favor chaoz更可能买
kindle(比起买电子琴with everything else equal)的prediction吗？
老model model的肯定不是sufficient statistic但可能抓住了某一方面的重点所以你
们公司会用，
anyway, what should you model?
【在 c***z 的大作中提到】
: 就是我们公司有一个model，预测网上shopping cart里面会有什么东西
: 我改进的model比老model RMSE要低
: 但是老model把cart size normalize to 1，虽然没什么依据
: 于是让我把新model也normalize，结果RMSE显示老model好
: 虽然我解释说因为normalization distorted data at local level所以RMSE不再是有
: 效的指示，并且用了US coast line 和 加州 coast line谁更长做例子，但是自己感觉
: 不是很hit the point
: 大家有什么idea么？谢谢啦
|
c********h
发帖数: 330 | 5 楼主能不能举个例子说明一下，没怎么看懂，可能是我没这方面背景 |
p****o
发帖数: 1340 | 6 what's your error metrics? weighting (I assume that's what the
renormalization does) might impact the performance assessment a lot.
【在 c***z 的大作中提到】
: 就是我们公司有一个model，预测网上shopping cart里面会有什么东西
: 我改进的model比老model RMSE要低
: 但是老model把cart size normalize to 1，虽然没什么依据
: 于是让我把新model也normalize，结果RMSE显示老model好
: 虽然我解释说因为normalization distorted data at local level所以RMSE不再是有
: 效的指示，并且用了US coast line 和 加州 coast line谁更长做例子，但是自己感觉
: 不是很hit the point
: 大家有什么idea么？谢谢啦
|
v*******e
发帖数: 11604 | 7 normalize之后当然RMSE就变了，residue分布已经不是normal分布了(其实原来就不是
normal分布，原来是poisson分布吧？)，既然不是normal分布，RMSE就不能做为一个分
辨好坏的指标。 |
A*******s
发帖数: 3942 | 8 这个问题应该是怎么把square error loss function和model实际目的联系起来。
按照OP的做法，直接将# of items作target variable然后minimize RMSE，其实更合理
的假设是target variable服从的是Poisson，或者至少应该假设是variance increases
with mean,那么optimize RMSE的后果是# of items比较大的observations会dominate
model。
按照old model的做法，把percentage作为target variable，这样optimize RMSE的后
果就刚好相反了，# of items比较小的observations会dominate model。
哪个更合理，就看business的需要了。 |
O*********h
发帖数: 140 | 9 这句话的更像是说：用户浏览每n个item，就把1个放入cart。听起来与你们公司的要求
不符啊。
【在 c***z 的大作中提到】
: My model (a decision tree) outperforms the old model if the unit of analysis
: is items bought
: (it should, since the old model predicts that everything the person viewed
: is bought)
: the old model outperforms when we fix the cart size to be 1 (the old model
: then predicts that 1/n of each item viewed is bought, where n = number of
: items viewed)
: I am not comfortable about fixing the cart size to be 1 at the first place...
|
c***z
发帖数: 6348 | 10 Thank you all so much for the inputs! As always, you guys are most helpful!
For some more context, we are trying to predict conversion based on page
view. The old model says everything will be bought, and then normalize so
that the cart size is one.
The new tree model tries to predict individual conversion rate. I played
with the tree model a little so that it outperforms the old model even after
normalizing.
I think it is the weighting (normalization) that distorted the data and RMSE.
I agree that RMSE is not perfect and the best way is to compare final data
such as market share with real data. However we are not confident about the
quantities yet (just the yes/no about purchase).
The next step is definitely to include more features into the model, as well
as use a output that is closer to the final product.
Please shoot any additional question and I will be very glad to discuss. |
|
|
c***z
发帖数: 6348 | 11 not multiclass, just a yes/no of purchase decision on each item viewed
and the old model predicts that each item will be bought, say, there are 5
of them; the model then say 0.2 of each item is bought
【在 A*******s 的大作中提到】
: 没看明白
: “shopping cart里面会有什么东西”这个算怎样的random variable？multiclass？
: 如果是multiclass的又怎么用RMSE来衡量？
|
c***z
发帖数: 6348 | 12 you nailed it, we are only working on yes/no now
we are using the old model because there was no statistician before me...
【在 r*****d 的大作中提到】
: "预测网上shopping cart里面会有什么东西"还不够精确，要看你们究竟对什么感兴趣
: ，才能建立loss function比较model.
: 举个例子，chaoz的cart里有两个kindle一个电子琴。如果我的预测是with 100%
: probability chaoz的cart里有一个kindle一个电子琴，那么loss function应该扣分吗
: ？如果不扣分那就说明你们只对yes/no感兴趣对数量并不感兴趣，任何不感兴趣的信息
: 最好不要出现在prediction里，因为presumably能简化model而又抓住了重点。
: 继续chaoz的cart里有两个kindle一个电子琴的例子，你们会favor chaoz更可能买
: kindle(比起买电子琴with everything else equal)的prediction吗？
: 老model model的肯定不是sufficient statistic但可能抓住了某一方面的重点所以你
: 们公司会用，
|
c***z
发帖数: 6348 | 13 yeah, that was my intuition, but I couldn't nail it exactly how the impact
happened
error metric was RMSE
I am considering confusion matrix, but the conversion is kind of sparse
events, so maybe something more sophisticated is needed
【在 p****o 的大作中提到】
: what's your error metrics? weighting (I assume that's what the
: renormalization does) might impact the performance assessment a lot.
|
c***z
发帖数: 6348 | 14 poisson would be a good idea, or some kind of truncated model
I am thinking about this for next step (quantities)
【在 v*******e 的大作中提到】
: normalize之后当然RMSE就变了，residue分布已经不是normal分布了(其实原来就不是
: normal分布，原来是poisson分布吧？)，既然不是normal分布，RMSE就不能做为一个分
: 辨好坏的指标。
|
c***z
发帖数: 6348 | 15 Yeah, I think you are exactly right!
increases
dominate
【在 A*******s 的大作中提到】
: 这个问题应该是怎么把square error loss function和model实际目的联系起来。
: 按照OP的做法，直接将# of items作target variable然后minimize RMSE，其实更合理
: 的假设是target variable服从的是Poisson，或者至少应该假设是variance increases
: with mean,那么optimize RMSE的后果是# of items比较大的observations会dominate
: model。
: 按照old model的做法，把percentage作为target variable，这样optimize RMSE的后
: 果就刚好相反了，# of items比较小的observations会dominate model。
: 哪个更合理，就看business的需要了。
|
c***z
发帖数: 6348 | 16 sorry, the old model says each item is bought 1/n units
【在 O*********h 的大作中提到】
: 这句话的更像是说：用户浏览每n个item，就把1个放入cart。听起来与你们公司的要求
: 不符啊。
|
g******2
发帖数: 234 | 17 does the value of a conversion make a difference? e.g. camera vs pencil. If
yes, then value weighting will be better weighting scheme. Assuming your
model is feeding to merchant ranking, # of converted items is highly gamable
. |
c***z
发帖数: 6348 | 18 yes, product category should definitely enter the model, but we are not
there yet...
If
gamable
【在 g******2 的大作中提到】
: does the value of a conversion make a difference? e.g. camera vs pencil. If
: yes, then value weighting will be better weighting scheme. Assuming your
: model is feeding to merchant ranking, # of converted items is highly gamable
: .
|
