g*****g
发帖数: 34805 | 1 Cassandra doesn't support high consistency. It doesn't support Atomicity
beyond single row. And on single row it can has false negative, namely a
failed write can write to the storage on partition. I am a big fan of C* but
let's be realistic here, it doesn't beat CAP.
design |
|
b*****n
发帖数: 618 | 2 quorum system本身就不是A的。
Cassandra tunnable consistency也不是你说的这种C，tune来tune去的结果是只能让
得到的结果是正确的概率更高而已，小数点后面几个9的差别。跟你想要的C根本不是一
回事。
A这个东西其实是你能接受的latency到底有多少，可以完全在有machine failure的这
段时间停止process任何request，然后等到完全恢复之后再继续，如果无限大的
latency还可以被认为是A，那就能实现A。关键是看这种cost有多大，能不能接受。
分布系统处理failure困难的一个地方就在于，在跟remote发信息的时候如果没收到ack
，不能分辨出到底是remote挂了还是network partition的，所以如果想要所谓的C，A
跟P基本上不能共存。Kafka design的时候有一个重要的假设，就是所有机器在同一个
network里面，假设不会出现partition。
即使是所谓的CP，CA系统，也不是完全100%的CP，CA，总有个别的case能break，只是
侧重点在什么上面，说来说去其实都是tra... 阅读全帖 |
|
t*********r
发帖数: 387 | 3 > Cassandra tunnable consistency也不是你说的这种C，tune来tune去的结果是只能
让得到的结果是正确的概率更高而已，小数点后面几个9的差别。跟你想要的C根本不是
一回事。
你确定跑个QUORUM也不能达到我所说的C？来个例子举证。
当然所有机子挂了我也没办法。
> A这个东西其实是你能接受的latency到底有多少，可以完全在有machine failure的
这段时间停止process任何request，然后等到完全恢复之后再继续，如果无限大的
latency还可以被认为是A，那就能实现A。
理论上A跟LATENCY没半毛钱关系。我不喜欢胡扯，直接引用文献：http://webpages.cs.luc.edu/~pld/353/gilbert_lynch_brewer_proof.pdf
Available Data Objects
For a distributed system to be continuously available, every request
received by a non-failing node in the sy... 阅读全帖 |
|
t*********r
发帖数: 387 | 4 Kafka's ISR together with its f failure tolerance in f+1 nodes is a bad
guarantee, which is why data can be lost.
availability |
|
t*********r
发帖数: 387 | 5 It's not an issue if you just use it as a KV store -- that is what was asked
of originally.
The second issue is implementation specific. I looked closer in the
documentation, and conceded this is the case in implementation. But do you
agree that that if you take a base quorum protocol, you can get a C* system?
Ensure that you operate only on a majority, and you get A?
but |
|
g*****g
发帖数: 34805 | 6 A KV store is not a general purpose DB. Yes, KV store, when key is properly
distributed, can guarantee timeline consistency on key level and tolerate
some failures. But that's not equal to C in CAP. Next time tell me something
I don't already know.
asked
system? |
|
t*********r
发帖数: 387 | 7 C in CAP is atomicity/serializability. Are you sure we agree on that?
I don't see why you don't think a KV store cannot achieve this level of
guarantee.
properly
something |
|
g*****g
发帖数: 34805 | 8 It means atomic consistency, and when KV store cannot do that beyond single
key, it's certainly not C in CAP.
The CAP Theorem is based on three trade-offs, one of which is "atomic
consistency" (shortened to "consistency" for the acronym), about which the
authors note, "Discussing atomic consistency is somewhat different than
talking about an ACID database, as database consistency refers to
transactions, while atomic consistency refers only to a property of a single
request/response operation seq... 阅读全帖 |
|
t*********r
发帖数: 387 | 9 I fail to see how you derived the statement from your citation:
"when KV store cannot do that beyond single key, it's certainly not C in CAP
."
Apart from whether it is feasible or not, why do you assert that multi key
support a requirement for consistency?
single
single |
|
g*****g
发帖数: 34805 | 10 So I need to help you interpret what it means now? The database below means
traditional RDBMS. So C in CAP means AC in RDBMS. Can RDMBS support AC on
multiple rows? I am sorry you really disappoint me, I thought I could learn
something from you but all I figure out is you are lacking of basics. You
don't beat CAP but interpreting it differently from what people understand.
And it has a different meaning than the Atomic in ACID, as it subsumes the
database notions of both Atomic and Consistent."... 阅读全帖 |
|
t*********r
发帖数: 387 | 11 Hmmm, I don't remember that C includes AC, but I will concede I'm wrong in
this regard.
Nonetheless, I disagree with your statement that it is not possible to have
multikey transaction in KV stores in general.
You already agree that atomicity for each key is possible. To add multikey
transaction, consider the following design:
-For each transaction, sort the shards they touch into some order. This
ensures that transactions that touch the same keys will send their requests
to the corresponding sh... 阅读全帖 |
|
g*****g
发帖数: 34805 | 12 You could simply use Zookeeper to guarantee exclusivity to achieve
distributed transaction and the performance will go down to the knees. There
's a reason most KV stores don't even support rollback.
Once a transaction crosses shards, your write performance will be worse than
a RDBMS. It's just physics.
have
requests
in |
|
t*********r
发帖数: 387 | 13 Let's address the issues one at a time.
Do you concede that C and A are both achievable along with P with the
provision that during a partition, every shard has a surviving majority?
Bring it up if you disagree, but I don't think that performance is relevant
in the discussion of "can you achieve it?"
There
than |
|
g*****g
发帖数: 34805 | 14 I feel you are keeping patching your definition of CAP along with your DB
design. And you concede your best design is deeply flawed with no
performance to begin with.
Performance issue means slow access, and slow access means low availability.
If all your clients try to write the same keys, does it scale? C* will do
to a certain extent, with eventual consistency, and yours will not. Now that
's basics.
relevant |
|
t*********r
发帖数: 387 | 15 Performance: using ZK implies pessimistic CC, which IMO will probably
perform slightly worse in the general case.
I concede cross shard transactions will incur a latency cost, but scaling
RDBMS to very large scales is a pain in the ass. For smaller data sets,
crossing a few shards isn't going to be *that* bad.
I never made an argument that there is no tradeoff between C/A against *
performance*. What I do have a problem is claiming that you can't have both
C and A (since P is widely claimed and ... 阅读全帖 |
|
t*********r
发帖数: 387 | 16 I try to stick with the definitions as defined by the paper. For the case
previously, I've conceded it as a mistake and adjusted my presentation
accordingly.
Availability is a property. You have it, or you don't. It doesn't come for
free.
Do you agree with this definition:
"in some ways this is a weak definition of availability: it puts no bound on
how long the algorithm may run before terminating, and therefore allows
unbounded computation."
If all your client tries to to write to the same key,... 阅读全帖 |
|
g*****g
发帖数: 34805 | 17 Availability is pretty well defined. It's pretty much equivalent to linear
scalability. With more traffic you can maintain latency with more machines,
you have high availability.
And I said keys, not key. C* is susceptible to hot row issues. But generally
you can design around it, the end result is that many clients to write to a
set of rows can still scale. In your case, the set is reduced to one row
once you support transaction, and the effect is very obvious.
on |
|
t*********r
发帖数: 387 | 18 Please point me to a reference that defines availability the way you have
described in the context of CAP.
My understanding is the following:
"For a distributed system to be continuously available, every request
received by a non-failing node in the system must result in a response [...]
this is a weak definition of availability: it puts no bound on how long the
algorithm may run before terminating, and therefore allows unbounded
computation."
From the same proof paper before. I'm also under the... 阅读全帖 |
|
g*****g
发帖数: 34805 | 19 Honestly I only care what the industry wants to achieve, and not how to
define the terms to prove one's right or wrong. And in this industry high
availability includes, but not limited to linear scalability.
.]
the |
|
|
t*********r
发帖数: 387 | 21 This is bullshit.
When I make a mistake I concede my errors openly and objectively. When you
make a mistake you make excuses. You were citing the same source earlier,
and now you go back on it.
If you said something like "there is a tradeoff between consistency and
performance/linear scalability," I might think about it for a bit and then
agree with you. But you were ranting earlier about how disappointed you were
when I got my terms wrong, and how you think I don't know even the basics.
Now you... 阅读全帖 |
|
g*****g
发帖数: 34805 | 22 You come up with a product that beats CAP, you will be billionaire. You come
up with a peer review paper that does it. You don't need to worry about
next job. It's your duty to prove yourself when you have a such bold claim.
I can be totally wrong and that doesn't prove you beat CAP, and we can agree
to disagree, so what are you trying to achieve here?
were
. |
|
t*********r
发帖数: 387 | 23 Because this is a solved problem and is not particularly interesting at this
point in academia or industry. I've
presented an argument outlining why the CAP tradeoff is not a universal
requirement if you're willing to compromise on latency and maintain specific
failure requirements. I've linked you a paper that achieves the three
properties as defined by peer reviewed definitions, and yet you still assert
I have not provided sufficient proof. It's sad to see you resort to ad
hominem arguments at... 阅读全帖 |
|
g*****g
发帖数: 34805 | 24 The paper repeats many times it didn't beat CAP theorem. If during partition
, a minority cluster can retain write and come out with integrity after
union, then that's something. But it simply says this cannot achieve
majority and it'll stop writing. i.e. Losing write availability during
partition.
It's not a wrong design, but it's not a new design. I guess that's why they
never became mainstream and had an early exit.
this
specific
assert |
|
t*********r
发帖数: 387 | 25 I really wish you follow what I've written, but I'll clarify anyway.
First, while the whitepaper has a good solution, their definition of A is
likewise incorrect. Reconsider their approach with the peer reviewed
definition.
Secondly, I've stated before that only the majority partition should proceed
during a partition. The minority N in a 2N+1 configuration should inform
the client to redirect to the majority. This process neither violates C, A (
note only a response is required, not necessarily... 阅读全帖 |
|
g*****g
发帖数: 34805 | 26 I really wish you can just give a solid presentation, and not this with 10
patches will work. If Foundation DB s official website didn't say they beat
it, Id rather just believe it as is and not to believe one more patch will
do wonder. Don't forget it can be 3 split and no write can be done at all.
proceed
(
formal |
|
a***n
发帖数: 623 | 27 学点mongo、redis、cassandra也不错，很多公司都用这几个做db或者cache。 |
|
t********5
发帖数: 522 | 28 cache这个如果换做我的话我可能会先山寨一个memcached，然后高级点再山寨一个
redis，再高级点可能要山寨cassandra，不过最后这个只知道大概没有实际用过
真让我说到这一步我还真就只能胡扯了…… |
|
s******y
发帖数: 936 | 29 西雅图startup 现有成员< 10,急需扩大工程师队伍，地点可以选西雅图或者上海（非
常欢迎去上海，总部将设在上海）。
我们使用的主要技术包括： Java, Nosql(MongoDB), Redis, Python, Nodejs,
AngularJS, Restful, Java 多线程（传统多线程技术和AKKA多线程）。。。。。
需要全职前台，后台工程师各若干，有兴趣或者任何问题请联系： aaronliang.job@
gmail.com |
|
d******e
发帖数: 2265 | 30 这个你怎么可能写类。
都是用现有工具。简单的redis，或者其他kv storage存信息。不要效率数据库好了。
B上线握手后，服务端一个actor开始push信息到设备上。结束。 |
|
d******e
发帖数: 2265 | 31 真实场景：比如twitter又数千台redis。做consistent hashing.一台坏了还有其他的。
又有很多台connection server比如说websocket活着其他tcp, udp，如果它们都跪了，
你也没办法。
而且server当了，数据还在，等重启再说呗。
设备1 of B登录后，通知server，我拿到第n条信息， server就起一个worker，可以是
一个erland actor,活着一个thread, 开始push信息到serve.活着pull也可以。
设备2 of B同理。
你可以想象一下weixin怎么做的。 |
|
p*****2
发帖数: 21240 | 32 ha和resharding都很麻烦
latency要求不特别高可上cassandra |
|
z****e
发帖数: 54598 | 33
这不是什么好迹象
cassandra毕竟是自己build from scratch
如果能成功，说明程序猴有造轮子的水平，属于大牛，可遇不可求
但是做失败了，说明水平达不到，也可以理解
毕竟不是阿猫阿狗都有造轮子实力的
那就用开源吧，没有造轮子的实力，有用开源轮子的实力
也不错，netflix就用了不少开源轮子，可惜hbase也用不好
现在开始转向enterprise products
zippydb就是enterprise products，是enterprise redis
后面一旦lockin，马上就是被插管吸血的节奏
搞不好离裁员不远了，股票逢高可以出一点了 |
|
z****e
发帖数: 54598 | 34
nosql
s
nah
zippydb is enterprise redis
u may check their website for more details
they even provide u the pricing of using it |
|
f*******t
发帖数: 7549 | 35 不知道我之前的回帖为啥没发出来。fb的zippydb是rocksdb+sharding，跟redis无关，
重名而已 |
|
z****e
发帖数: 54598 | 36
nosql会用的越来越多，就跟你用hibernate一样
虽然你不是dba，但是你总得会点怎么连数据库，建连接池，crud，orm这些事情吧？
cassandra搞一搞咯，还有redis之类的cache
然后streaming，看看rxjava |
|
H*****s
发帖数: 9 | 37 如果iterator可以按photoid排序，那一个heap就可以，
否则用hashmap存photoid和count
如果单机内存不够放整个hashmap，可以考虑用distributed memory cache，比如
memcache或者redis.
存数据的时候最好用photoid shard到不同的小文件中，这样计算的时候不需要存整个
photoid的空间，可以把计算top m的工作distribute到多台机器上计算，最后merge。
也可以用mapreduce |
|
j**********3
发帖数: 3211 | 38 请教一下各位大牛！
什么时候用mysql/psql， 什么时候用mongodb，什么时候用redis？ |
|
a*****u
发帖数: 1712 | 39 从实用角度，流量大的时候谨慎使用mongodb，需要persistent存储的时候谨慎使用
redis。 |
|
j**********3
发帖数: 3211 | 40 对于mongo，我查了一下，以前也听朋友说，是因为不安全。
为啥“流量大”的时候不能用mongodb？
redis为啥persistent的时候不能用？
大牛请为菜鸟详解 |
|
R*********n
发帖数: 99 | 41 Dears,
We're looking for Senior PHP Developer (Several) in Shanghai, China for a US
company which is setting up a team there.
Senior PHP Developer
Job Summary:
The Developers work closely with design and product teams to implement
modern, clean and effective web user interfaces in multiple high-traffic web
properties.
Location: Shanghai
Requirements:
* Expert level PHP, experience with PHP framework, Zend Framework,
CodeIgniter, Yii etc.
* Experience with PHP CMS system, Wordpress, Drupal etc.
*... 阅读全帖 |
|
d******e
发帖数: 2265 | 42 R3:这个搞AKKA，stream都可以搞定。
R2: 很多吧。redis， memcached都是。
你可以timedout。等等。可以lru一个memory chunk. |
|
d******e
发帖数: 2265 | 43 R3:这个搞AKKA，stream都可以搞定。
R2: 很多吧。redis， memcached都是。
你可以timedout。等等。可以lru一个memory chunk. |
|
j****i
发帖数: 36 | 44 地点，windward。总公司研发在中国，需要招2个在美国工作的基于AWS的云工程师，必
须讲中文，有绿卡。请感兴趣者联系，[email protected]
/* */
Cloud Deployment & Operations Engineer
Come join an exciting and fast-moving team in the fast growing area of the
home automation of the IoT world. We are looking for a highly motivated
individual who possesses deep engineering expertise to help design/operate/
enhance/troubleshoot a cloud platform that makes the connected home a
reality, who can demonstrate excellent communication skill (written and
verb... 阅读全帖 |
|
p*u
发帖数: 2454 | 45 实习求职终于uber offer，全职求职始于uber rej，what a life。带着唯一的return
offer继续骑驴找马找全职了。
====感想====
0，千万不要看点科技博客，小文章就觉得懂design了，千万不要！常见design题有什
么用，还是被大牛们按在地上摩擦了！所以下面都是我自己总结出来的合适答案，可能
还是会导致你被按在地上摩擦，大家随便看看。
1，Uber NY的Bar很高，尤其是design bar，不想浪费机会的小伙伴还是申Uber SF吧。
刷算法基本对Uber NY没用的，目前Uber NY就没有new grad，第一批new grad的要求必
然是overflow的。
2，几乎纯Design面试，new grad永远的痛，狠狠一巴掌打在自以为design很熟的我脸
上。面我的四个人都干过小公司CTO甚至大公司的技术核心，“你们这些说名词的new
grad，根本不懂design的境界”。
3，真机IDE测试，能bug free就bug free，对方的笑容不代表他认可你先run再debug。
大家都是专业的，笑容什么都不代表。
4，Uber N... 阅读全帖 |
|
f*******r
发帖数: 976 | 46 Move on.
实习求职终于uber offer，全职求职始于uber rej，what a life。带着唯一的return
offer继续骑驴找马找全职了。
====感想====
0，千万不要看点科技博客，小文章就觉得懂design了，千万不要！常见design题有什
么用，还是被大牛们按在地上摩擦了！所以下面都是我自己总结出来的合适答案，可能
还是会导致你被按在地上摩擦，大家随便看看。
1，Uber NY的Bar很高，尤其是design bar，不想浪费机会的小伙伴还是申Uber SF吧。
刷算法基本对Uber NY没用的，目前Uber NY就没有new grad，第一批new grad的要求必
然是overflow的。
2，几乎纯Design面试，new grad永远的痛，狠狠一巴掌打在自以为design很熟的我脸
上。面我的四个人都干过小公司CTO甚至大公司的技术核心，“你们这些说名词的new
grad，根本不懂design的境界”。
3，真机IDE测试，能bug free就bug free，对方的笑容不代表他认可你先run再debug。
大家都是专业的，笑容什么都不代表... 阅读全帖 |
|
g*****g
发帖数: 34805 | 47 扔进C*，最多上面再加一层memcache/redis就是了。 |
|
o******0
发帖数: 105 | 48 如果工作中没用到过kafka,redis, storm/spark之类的热门东西，面试design题时，说
要用它们解决，能行吗？问深了，会不会露馅？ |
|
d****3
发帖数: 5 | 49 公司40多年安全监控领域背景，但是规模不大，主要是作为服务供应商。新项目今年上
马，公共安全领域，客户主要是政府和学校。现在需要至少两名软件工程师，历史原因
使用微软平台，C＃，SQLSERVER，基本这个行业都这样， 我也没办法， 但是会有大量
机会接触elasticsearch， redis，aws等。懂.Net, ASP MVC优先,如果有大数据经验加
分，同时非常欢迎应届的同学，我也是这么过来的，知道大家不容易，本科研究生不限
，同时办身份，1－2年绿卡。中西部，待遇肯定无法和湾区比，本地比上不足比下有余
。我是项目负责人，有兴趣站内我。 |
|
d****3
发帖数: 5 | 50 elasticsearch, redis不是必须，但是将来一定要用。.net，sqlserver，EF，mvc多少
要求会一些，这些学起来也不难。只是害怕提到微软的一套技术大家看不上，被鄙视。
但是visual studio真心做得好。这个月开始参加各种展会推广业务，
估计年底正式开始找人。
邮箱：e24jobs "at" gmail |
|
