g*****g
发帖数: 34805 | 1 我试图多线程处理一个Queue上的数据，数据是一个email address和一些相关
数据。email address作为用户名，在DB里必须是unique的。我多线程处理
这些用户的时候，产生的问题是多个线程都试图建立同一个用户，这时候
后commit的线程就会出错。怎么才可以避免这个问题而又尽量保持性能呢？
我用的是hibernate和spring. |
A**o
发帖数: 1550 | 2 get an email based schedule? |
F****n
发帖数: 3271 | 3 保持性能的意思是不能有共享区？否则很容易解决啊。
【在 g*****g 的大作中提到】
: 我试图多线程处理一个Queue上的数据，数据是一个email address和一些相关
: 数据。email address作为用户名，在DB里必须是unique的。我多线程处理
: 这些用户的时候，产生的问题是多个线程都试图建立同一个用户，这时候
: 后commit的线程就会出错。怎么才可以避免这个问题而又尽量保持性能呢？
: 我用的是hibernate和spring.
|
a****i
发帖数: 1182 | 4 一个想法是把queue分块，比方说线程一处理0到9，线程二处理10到19
或者用Hibernate直接saveorupdate 不知道行不行
【在 g*****g 的大作中提到】
: 我试图多线程处理一个Queue上的数据，数据是一个email address和一些相关
: 数据。email address作为用户名，在DB里必须是unique的。我多线程处理
: 这些用户的时候，产生的问题是多个线程都试图建立同一个用户，这时候
: 后commit的线程就会出错。怎么才可以避免这个问题而又尽量保持性能呢？
: 我用的是hibernate和spring.
|
g*****g
发帖数: 34805 | 5 I am thinking something like this
if(user doesn't exist) {
synchronize(address.intern()) {
transaction begins;
if(user doesn't exist) {
create user
}
transaction ends;
}
}
// process user
Any better idea?
【在 g*****g 的大作中提到】
: 我试图多线程处理一个Queue上的数据，数据是一个email address和一些相关
: 数据。email address作为用户名，在DB里必须是unique的。我多线程处理
: 这些用户的时候，产生的问题是多个线程都试图建立同一个用户，这时候
: 后commit的线程就会出错。怎么才可以避免这个问题而又尽量保持性能呢？
: 我用的是hibernate和spring.
|
c*****t
发帖数: 1879 | 6 This is probably the best you can get, which is the check-and-check-again
paradigm.
Depending on your collision frequency, other approaches may exist. Mostly
dealing with some sort of cache.
【在 g*****g 的大作中提到】
: I am thinking something like this
: if(user doesn't exist) {
: synchronize(address.intern()) {
:
: transaction begins;
: if(user doesn't exist) {
: create user
: }
: transaction ends;
: }
|
A**o
发帖数: 1550 | 7 roll back (delete) user if process failed.
【在 g*****g 的大作中提到】
: I am thinking something like this
: if(user doesn't exist) {
: synchronize(address.intern()) {
:
: transaction begins;
: if(user doesn't exist) {
: create user
: }
: transaction ends;
: }
|
g*****g
发帖数: 34805 | 8 transaction would be declarative by spring and rollback
should be automatic. It's transaction boundary that should
be careful.
【在 A**o 的大作中提到】
: roll back (delete) user if process failed.
|
k***r
发帖数: 4260 | 9 Use hash value to partition users to a specific thread?
【在 g*****g 的大作中提到】
: 我试图多线程处理一个Queue上的数据，数据是一个email address和一些相关
: 数据。email address作为用户名，在DB里必须是unique的。我多线程处理
: 这些用户的时候，产生的问题是多个线程都试图建立同一个用户，这时候
: 后commit的线程就会出错。怎么才可以避免这个问题而又尽量保持性能呢？
: 我用的是hibernate和spring.
|
g*****g
发帖数: 34805 | 10 That doesn't sound right. I don't want to create a new
user (which will result an error), but I do want to create
some new records that are associated with the user in
each thread. And I want them to be created as fast as possible.
Let's say if you have 10 messages coming in all for a new
user. You want to create a user record, and 10 message records
associated with this user. Obviously if you have 10 threads.
The fastest way is to create a user and commit, then all 10
threads create message rec
【在 k***r 的大作中提到】
: Use hash value to partition users to a specific thread?
|
k***r
发帖数: 4260 | 11 I assumed the DB was the bottleneck in a lot of cases so
having multiple threads hitting the DB wouldn't help. I also
assumed that the messages coming in were somewhat random
in terms of user name. However both of my assumption could
be false, and it sounds like they are false in your case.
If this is the case, the double-checked lock pattern seems
to be the way to go.
【在 g*****g 的大作中提到】
: That doesn't sound right. I don't want to create a new
: user (which will result an error), but I do want to create
: some new records that are associated with the user in
: each thread. And I want them to be created as fast as possible.
: Let's say if you have 10 messages coming in all for a new
: user. You want to create a user record, and 10 message records
: associated with this user. Obviously if you have 10 threads.
: The fastest way is to create a user and commit, then all 10
: threads create message rec
|
m******t
发帖数: 2416 | 12 So create the user first, then you are free to do the rest concurrently
and independently without synchronization.
【在 g*****g 的大作中提到】
: That doesn't sound right. I don't want to create a new
: user (which will result an error), but I do want to create
: some new records that are associated with the user in
: each thread. And I want them to be created as fast as possible.
: Let's say if you have 10 messages coming in all for a new
: user. You want to create a user record, and 10 message records
: associated with this user. Obviously if you have 10 threads.
: The fastest way is to create a user and commit, then all 10
: threads create message rec
|
F****n
发帖数: 3271 | 13 他是想把所以的都放到SPRING的Declaration里吧，否则这个问题很好解决，在
Transaction之前先访问CACHE，如果ID已经存在抱错。进入TRANSACTION后把ID写入
CACHE中。这样如果有10个线程创立同一ID，只有一个需要进入DB TRANSACTION。
【在 m******t 的大作中提到】
: So create the user first, then you are free to do the rest concurrently
: and independently without synchronization.
|
m****i
发帖数: 302 | 14 assign user name to blocks, e.g.: only need to check A-C block when you got
a message from Alice.
【在 k***r 的大作中提到】
: Use hash value to partition users to a specific thread?
|
