我正在尝试制作一个具有多线程的网络爬虫以使其更快。我想让每次执行都增加价值。但有时该值会跳过或重复自身。import threadingnum = 0def scan(): while True: global num num += 1 print(num) open('logs.txt','a').write(str(f'{num}\n'))for x in range(500): threading.Thread(target=scan).start()结果：22557810101213131316171919222324252628292931323334预期结果：12345678910
3 回答
犯罪嫌疑人X
TA贡献2080条经验 获得超4个赞
因此，由于该变量num是共享资源，因此您需要对其进行锁定。这是按如下方式完成的：
num_lock = threading.Lock()
每次要更新共享变量时，都需要线程先获取锁。一旦获得锁，只有该线程可以更新 num 的值，而在当前线程获得锁时，其他线程将无法这样做。
确保在执行此操作时使用waitortry-finally块，以保证即使当前线程无法更新共享变量也将释放锁。
是这样的：
num_lock.acquire()
try:
num+=1
finally:
num_lock.release()
使用with：
with num_lock:
num+=1
当年话下
TA贡献1890条经验 获得超9个赞
一个重要的标注，除了threading.Lock：
用于
join使父线程等待分叉线程完成。没有这个，线程仍然会竞争。
假设我正在使用num线程完成后：
import threading
lock, num = threading.Lock(), 0
def operation():
global num
print("Operation has started")
with lock:
num += 1
threads = [threading.Thread(target=operation) for x in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
print(num)
没有连接，不一致（9 被打印一次，否则 10）：
Operation has started
Operation has started
Operation has started
Operation has started
Operation has startedOperation has started
Operation has started
Operation has started
Operation has started
Operation has started9
使用 join，它是一致的：
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
Operation has started
10
梵蒂冈之花
TA贡献1900条经验 获得超5个赞
似乎是一种竞争条件。您可以使用锁，以便只有一个线程可以获得特定的数字。使用 lock 写入输出文件也是有意义的。
这是一个带锁的例子。当然，您不能保证输出的写入顺序，但每个项目都应该恰好写入一次。在此示例中，我添加了 10000 的限制，以便您可以更轻松地检查最终是否将所有内容写入测试代码，因为否则无论您在什么时候中断它，都很难验证数字是被跳过还是只是等待锁定以写入输出。
不是my_num共享的，因此您已经在该with num_lock部分内声明了它之后，您可以自由释放该锁（保护共享num）然后继续在my_num外部使用with，而其他线程可以访问该锁以声明自己的价值. 这最大限度地减少了持有锁的持续时间。
import threading
num = 0
num_lock = threading.Lock()
file_lock = threading.Lock()
def scan():
global num_lock, file_lock, num
while num < 10000:
with num_lock:
num += 1
my_num = num
# do whatever you want here using my_num
# but do not touch num
with file_lock:
open('logs.txt','a').write(str(f'{my_num}\n'))
threads = [threading.Thread(target=scan) for _ in range(500)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
添加回答
举报
0/150
提交
取消