Repository navigation
Commit 7bb29b9
feat: env-var configurable httpx pool config in split_pdf_hook
Make httpx.AsyncClient pool config in run_tasks configurable via env vars:
- UNSTRUCTURED_CLIENT_MAX_CONNECTIONS (default 100)
- UNSTRUCTURED_CLIENT_MAX_KEEPALIVE_CONNECTIONS (default 20)
- UNSTRUCTURED_CLIENT_KEEPALIVE_EXPIRY (default 5.0 seconds)
Defaults match httpx's built-in defaults, so this is backward compatible.
Also extends the split_pdf event=plan_created INFO log to include the
resolved pool values so operators can verify the active config from
production logs.
Motivation: when the SDK runs inside a Kubernetes Deployment that
load-balances only at TCP-connect time (kube-proxy ClusterIP, meshless),
httpx's default keepalive pooling locks onto a small subset of backend
pods. Allowing operators to lower MAX_KEEPALIVE_CONNECTIONS and
KEEPALIVE_EXPIRY forces frequent reconnects and redistributes traffic.
Encountered on INC-11422.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>1 parent 5e6f238 commit 7bb29b9
2 files changed
Lines changed: 72 additions & 3 deletions
File tree
- _test_unstructured_client/unit
- src/unstructured_client/_hooks/custom
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
482 | 482 | | |
483 | 483 | | |
484 | 484 | | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
485 | 523 | | |
486 | 524 | | |
487 | 525 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
180 | 202 | | |
181 | 203 | | |
182 | 204 | | |
| |||
205 | 227 | | |
206 | 228 | | |
207 | 229 | | |
| 230 | + | |
| 231 | + | |
208 | 232 | | |
209 | | - | |
| 233 | + | |
210 | 234 | | |
211 | 235 | | |
212 | 236 | | |
213 | 237 | | |
214 | 238 | | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
215 | 242 | | |
216 | 243 | | |
217 | | - | |
| 244 | + | |
218 | 245 | | |
219 | 246 | | |
220 | 247 | | |
| |||
770 | 797 | | |
771 | 798 | | |
772 | 799 | | |
| 800 | + | |
773 | 801 | | |
774 | | - | |
| 802 | + | |
775 | 803 | | |
776 | 804 | | |
777 | 805 | | |
| |||
790 | 818 | | |
791 | 819 | | |
792 | 820 | | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
793 | 824 | | |
794 | 825 | | |
795 | 826 | | |
| |||
0 commit comments