[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Large number of SQL Connection from airflow scheduler

Hi Anand,
>From my experience that reusing connection pool across those parsing
processes would be dangerous( recall I got some connection corruption error
because of that). If you find that to be too many connection you can
probably tune down the max_threads
reduce the number of parsing processes or make the scheduler parse slower
by tune down min_file_process_interval

Kevin Y

On Thu, Dec 6, 2018 at 5:53 PM anandgupta1412@xxxxxxxxx <
anandgupta1412@xxxxxxxxx> wrote:

> HI,
> I have a query. It will be great if somebody in the community can clarify
> it.
> Basically, for each dag_processing process, I can see the orm being
> reconfigured and disposed towards the end. This is creating and closing a
> lot of connections on MySQL server, which crashes in scenarios where there
> are a lot of new dags to process.
> I want to know if this is the intended behavior and why? Also what kind of
> improvement can be planned in this area so that we can reuse the connection
> pool amongst these processes.
> Thanks,
> Anand