Categories
Mastering Development

Flask Application with multiprocessing randomly connection refused with multi request

I have an API written in a flask that crawls data from the site using selenium. when making a single request it works fine. but, when I make multiple requests from my local then gives connection refused error for some requests.

app.py

app = Flask(__name__)

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=5001, debug=False)

when I make a request through the flask server it manages 4 processes of multiprocessing and gives response accordingly. when I make only one request then it’s working fine and give a response from all 4 processes of multiprocess. but, when I make multiple requests at a time then it gives an error as connection refused for some random processes. and I’m not getting a response from all 4 processes.

def multiprocess_method():
manager = multiprocessing.Manager()

# result return dict of al  process
return_dict = manager.dict()
log_obj_dict = manager.dict()

# start first process of combination false, false
p1 = Process(target=crawl,args=(return_dict,log_obj_dict))
p1.start()
time.sleep(0.1)

# start second process of combination false, true
p2 = Process(target=crawl,args=(return_dict,log_obj_dict))
p2.start()
time.sleep(0.1)

# start third process of combination true, true
p3 = Process(target=crawl,args=(return_dict,log_obj_dict))
p3.start()
time.sleep(0.1)

# start fourth process of combination true, false
p4 = Process(target=crawl,args=(return_dict,log_obj_dict))
p4.start()
time.sleep(0.1)

process_list = [p1,p2,p3,p4]

while True:
    time.sleep(0.01)
    if not any(process.is_alive() for process in process_list):
        for process in process_list:
            process.join()
        return return_dict, log_obj_dict

here, crawl method which is a target of multiprocess which performs some actions. my problem is when I make multiple requests at a time it’s give me an error like connection refused otherwise for only one request at a time it’s working good.

Leave a Reply

Your email address will not be published. Required fields are marked *