Skip to content

Bulk load connection timeouts on 1.7.1 #595

Closed

Description

Describe the bug

As soon as I updated to 1.7.1 (from 1.6.x) I started experiencing occasional errors when attempting to bulk load: "RuntimeErorr: Failed to connect to SQL Server: Timeout Error: Timeout while connecting"

I'm not really sure how to create a repro of the issue as it happened seemingly randomly. What I can say is that the code in question makes heavy use of multiprocessing, often running concurrent bulk loads on several processes at once. This setup has been rock solid on 1.6.x and I didn't change any other dependencies, so I'm very confident that this is a regression.

Further technical details

Python version: 3.12
SQL Server version: SQL Server 2022
Operating system: Windows 11

Activity

  1. changed the title [-]Bulk load connection timeouts on 1.7[/-] [+]Bulk load connection timeouts on 1.7.1[/+] on May 21, 2026
  2. github-actions commented on May 21, 2026

    @github-actions

    Hi Adam Machanic (@amachanic), thank you for opening this issue!

    Our team will review it shortly. We aim to triage all new issues within 24-48 hours and get back to you.

    If you have additional information to share, please feel free to update the issue.

    Thank you for your patience!

  3. amachanic commented on May 21, 2026

    @amachanic
    Author

    Update: I tested some more using the code below. Running the code on Windows 11 with mssql-python 1.7.1, I am able to consistently see a failure. (I had to bump around some of the numbers a bit, so this may depend on instance size.) If I run the same either with mssql-python 1.6.0 or either version on Linux (WSL, Ubuntu LTS), I have yet to see a failure.

    So, bottom line, this issue appears to be isolated to Windows, and it is definitely a mssql-python 1.7.x issue.

    import multiprocessing
    import mssql_python
    
    
    _CONN_STR = [put your connection string here]
    
    
    def t():
        conn = mssql_python.connect(_CONN_STR, autocommit=True, timeout=10)
        cur = conn.cursor()
    
        cur.bulkcopy(
            "##test", ((r, r, r) for r in range(100000)),
            timeout=1800
        )
    
    
    if __name__ == '__main__':
        conn = mssql_python.connect(_CONN_STR, autocommit=True, timeout=10)
        cur = conn.cursor()
    
        cur.execute("DROP TABLE IF EXISTS ##test; CREATE TABLE ##test(i int, j int, k int);")
    
        with multiprocessing.Pool(processes=32) as pool:
            pool.starmap(t, [() for _ in range(128)])
    
        cur.execute("SELECT COUNT(*) FROM ##test")
        print(cur.fetchall())
    
  4. amachanic commented on May 21, 2026

    @amachanic
    Author

    And followup question that just occurred to me: What exactly does bulkcopy use for its connection timeout? Is it inherited from the outer connection object? (Which would be my assumption.) Or is there some other number in there somewhere? I don't see any obvious property to directly manipulate it.

  5. saurabh500 commented on May 22, 2026

    @saurabh500
    Contributor

    Adam Machanic (@amachanic)
    I tried and just cannot reproduce the issue. I have tried various sizes of a Windows VM and with artificially injected latencies.

    I need help from you to collect traces since you have a consistent repro.

    Steps

    1. Use a clean working directory so the log files are easy to collect:
      mkdir C:\mssql595-trace
      cd C:\mssql595-trace
    2. Copy your repro script (repro.py, or whatever you named it) into C:\mssql595-trace.
    3. Enable tracing by setting two environment variables in the same PowerShell window you'll run the repro from:
      $env:MSSQL_TDS_TRACE = "true"
      $env:MSSQL_TDS_TRACE_LEVEL = "debug"
      Valid levels are error, warn, info, debug, trace. Start with debug; only escalate to trace if debug doesn't show the cause (it produces ~10× more data).
    4. Run the repro:
      python .\repro.py
      You'll see two lines on stderr per process, confirming tracing is active:
      [mssql-py-core] Created log file → C:\mssql595-trace\mssql_python_logs\mssql_tds_trace__.log
      [mssql-py-core] Tracing enabled
    5. Locate the failing worker. The RuntimeError: Failed to connect to SQL Server: ... timed out after 15000ms traceback includes the worker PID (or you can match by the parent's multiprocessing stack frame). Each process writes its own file named mssql_tds_trace__.log under C:\mssql595-trace\mssql_python_logs.
    6. Zip and attach the logs:
      Compress-Archive .\mssql_python_logs C:\mssql595-trace\logs.zip
      Then attach logs.zip to this issue. Files are CSV (timestamp, thread_id, LEVEL, target, message) so they're safe to inspect first if you want.

    Notes

    • *Output location is \mssql_python_logs* — that's why step 1 matters. Whatever directory you launch python from is where logs land.
    • One file per process. A 32-worker Pool produces ~33 files (1 parent + 32 workers). Workers are reused across tasks, so it won't be 128 files.
    • Expected size at debug level: ~1–5 MB per worker file for your 128-task run. Total ~30–150 MB before zipping. At trace level, expect ~10×.
    • The parent process file may be 0 bytes if main doesn't open a connection itself — that's normal, the interesting data is in the worker files.
    • Sensitive data: the trace logs the server hostname, encryption mode, database name, and connection parameters (excluding passwords/tokens). Please review before posting publicly, or send the zip privately if you prefer.
    • Don't leave tracing on permanently — MSSQL_TDS_TRACE=true adds disk I/O on every connection. Unset it (Remove-Item env:MSSQL_TDS_TRACE) when you're done.

    What we're looking for

    In the worker file that contains the failing PID, we'll match the timestamp of the 15-second timeout against the last logged event to identify whether the stall is in DNS resolution, TCP connect, TLS handshake, or LOGIN7. That'll tell us where to look next.

    If you are not comfortable attaching the logs to this bug repro, then you can mail the logs to me. My email address hint is in my Github profile page at https://github.com/saurabh500

  6. amachanic commented on May 26, 2026

    @amachanic
    Author

    Saurabh Singh (@saurabh500) Unfortunately, or fortunately, some updates were installed on my computer over the weekend and lo and behold it now doesn't fail anymore.

    I still think something is off; my entire local test suite (associated with the application I'm developing) finishes with 1.6.0 in ~3m40s, and takes just a hair over 4m if I upgrade to 1.7.1. I ran both of them a bunch of times today and it's consistent. I also tried reproducing that in isolation: Connecting a bunch of times, bulk copying a bunch of times ... and I can't quite figure out where the time is coming from. The test in question bulk copies thousands of times and otherwise runs tens of thousands of queries, and while everything is quite well logged it's a bit tough to pinpoint. But I'll try to keep looking at it as time permits.

    Anyway, I guess I'll close this ticket down unless you have any further ideas/thoughts.

    In any case, thanks for looking into it! Much appreciated.

  7. saurabh500 commented on Jun 4, 2026

    @saurabh500
    Contributor

    I am reopening this issue. Based on more investigation to see why Adam Machanic (@amachanic) faced these problems, there are some speculative areas of improvement in TLS on Windows. I want to use this issue to track the progress.

  8. saurabh500 commented on Jun 5, 2026

    @saurabh500
    Contributor

    test-wheels.zip

    Attaching sample wheels

  9. saurabh500 commented on Jun 5, 2026

    @saurabh500
    Contributor
  10. saurabh500 commented on Jun 10, 2026

    @saurabh500
    Contributor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area: bulk-copyIssues in cursor.bulkcopy()triage neededFor new issues, not triaged yet.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions