l*****c
发帖数: 1153 | 1 被大家吊起兴趣了，呵呵。
问个问题。Pythong内建的multi-core support如何？比如像map()这样在collection上
的operation，有内建的multithreading机制么？ |
r****t
发帖数: 10904 | 2 You are welcome to look into python. :)
There's multithreading, but you cannot utilize multi-core using one process,
no matter how many threads you have, because of GIL problem.
One has to look somewhere else if you are to use default CPython, like
processing module, or IPython1 to get parallelized map() operations.
Or use mpipython etc etc...
【在 l*****c 的大作中提到】
: 被大家吊起兴趣了，呵呵。
: 问个问题。Pythong内建的multi-core support如何？比如像map()这样在collection上
: 的operation，有内建的multithreading机制么？
|
l*****c
发帖数: 1153 | 3 3x. GIL is definitely ridiculous. Is there a 3rd party module utilize hadoop
or other grid mode for this?
process,
【在 r****t 的大作中提到】
: You are welcome to look into python. :)
: There's multithreading, but you cannot utilize multi-core using one process,
: no matter how many threads you have, because of GIL problem.
: One has to look somewhere else if you are to use default CPython, like
: processing module, or IPython1 to get parallelized map() operations.
: Or use mpipython etc etc...
|
r****t
发帖数: 10904 | 4 GIL has deep reasons that are still not coped with. You can use hadoop via
Pyton, from my impression.
hadoop
【在 l*****c 的大作中提到】
: 3x. GIL is definitely ridiculous. Is there a 3rd party module utilize hadoop
: or other grid mode for this?
:
: process,
|
l*****c
发帖数: 1153 | 5 That's not what I mean. We use hadoop with python. But that is hadoop
calling python.
This is the other way: use hadoop to implement map() and reduce() in python,
so python could be used as the master language.
【在 r****t 的大作中提到】
: GIL has deep reasons that are still not coped with. You can use hadoop via
: Pyton, from my impression.
:
: hadoop
|
r****t
发帖数: 10904 | 6 I dnnot know a map() implemented with hadoop.
But there's such things in IPython1. or maybe "processing" module that's
going to enter standard library since python2.6. If processing is in
standard library, there's hope to see a map() that makes use of SMP...
python,
【在 l*****c 的大作中提到】
: That's not what I mean. We use hadoop with python. But that is hadoop
: calling python.
: This is the other way: use hadoop to implement map() and reduce() in python,
: so python could be used as the master language.
|
l*****c
发帖数: 1153 | 7 OK. I read a little about IPython1. It uses MPI to implement the
architecture. Gonna to read more this weekend to see if it is worthy to
implement one to accommodate Hadoop, which I would expect to be much less
work and more handful than implement IPython1.
【在 r****t 的大作中提到】
: I dnnot know a map() implemented with hadoop.
: But there's such things in IPython1. or maybe "processing" module that's
: going to enter standard library since python2.6. If processing is in
: standard library, there's hope to see a map() that makes use of SMP...
:
: python,
|
r****t
发帖数: 10904 | 8 ?? It can work with MPI, but MPI is not required.
I am using it without MPI...
【在 l*****c 的大作中提到】
: OK. I read a little about IPython1. It uses MPI to implement the
: architecture. Gonna to read more this weekend to see if it is worthy to
: implement one to accommodate Hadoop, which I would expect to be much less
: work and more handful than implement IPython1.
|
l*****c
发帖数: 1153 | 9 I just read the introduction, maybe I missed something.
【在 r****t 的大作中提到】
: ?? It can work with MPI, but MPI is not required.
: I am using it without MPI...
|
r****t
发帖数: 10904 | 10 刚刚看见这个应该是你要的。不知道你那个是不是搞定了。
http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant
python,
【在 l*****c 的大作中提到】
: That's not what I mean. We use hadoop with python. But that is hadoop
: calling python.
: This is the other way: use hadoop to implement map() and reduce() in python,
: so python could be used as the master language.
|
h***z
发帖数: 233 | 11 Hadoop is in Java, right? Why not just try using Jython?
python,
【在 l*****c 的大作中提到】
: That's not what I mean. We use hadoop with python. But that is hadoop
: calling python.
: This is the other way: use hadoop to implement map() and reduce() in python,
: so python could be used as the master language.
|
