Skip to content

Instantly share code, notes, and snippets.

@arssher

arssher/gist:aabd2c0db7da2bf5c508db59ee8a6cfd

Created June 9, 2023 13:39
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save arssher/aabd2c0db7da2bf5c508db59ee8a6cfd to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save arssher/aabd2c0db7da2bf5c508db59ee8a6cfd to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
================================================================================= FAILURES =================================================================================
________________________________________________________________________ test_pgbench[neon-45-400] _________________________________________________________________________
self = <fixtures.pageserver.http.PageserverHttpClient object at 0x7f7b2bc5aad0>, res = <Response [500]>
def verbose_error(self, res: requests.Response):
try:
> res.raise_for_status()
test_runner/fixtures/pageserver/http.py:119:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Response [500]>
def raise_for_status(self):
"""Raises :class:`HTTPError`, if one occurred."""
http_error_msg = ""
if isinstance(self.reason, bytes):
# We attempt to decode utf-8 first because some servers
# choose to localize their reason strings. If the string
# isn't utf-8, we fall back to iso-8859-1 for all other
# encodings. (See PR #3538)
try:
reason = self.reason.decode("utf-8")
except UnicodeDecodeError:
reason = self.reason.decode("iso-8859-1")
else:
reason = self.reason
if 400 <= self.status_code < 500:
http_error_msg = (
f"{self.status_code} Client Error: {reason} for url: {self.url}"
)
elif 500 <= self.status_code < 600:
http_error_msg = (
f"{self.status_code} Server Error: {reason} for url: {self.url}"
)
if http_error_msg:
> raise HTTPError(http_error_msg, response=self)
E requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:15003/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline/1360a54b2bd92aa3e9d6c90d3404a976/checkpoint
../../.cache/pypoetry/virtualenvs/neon-tCbAJW9a-py3.11/lib/python3.11/site-packages/requests/models.py:1021: HTTPError
The above exception was the direct cause of the following exception:
neon_with_baseline = <fixtures.compare_fixtures.NeonCompare object at 0x7f7b2bc5bb10>, scale = 400, duration = 45
@pytest.mark.parametrize("scale", get_scales_matrix())
@pytest.mark.parametrize("duration", get_durations_matrix())
def test_pgbench(neon_with_baseline: PgCompare, scale: int, duration: int):
> run_test_pgbench(neon_with_baseline, scale, duration, PgBenchLoadType.INIT)
test_runner/performance/test_perf_pgbench.py:174:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test_runner/performance/test_perf_pgbench.py:97: in run_test_pgbench
init_pgbench(
test_runner/performance/test_perf_pgbench.py:38: in init_pgbench
env.flush()
test_runner/fixtures/compare_fixtures.py:137: in flush
self.pageserver_http_client.timeline_checkpoint(self.tenant, self.timeline)
test_runner/fixtures/pageserver/http.py:404: in timeline_checkpoint
self.verbose_error(res)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <fixtures.pageserver.http.PageserverHttpClient object at 0x7f7b2bc5aad0>, res = <Response [500]>
def verbose_error(self, res: requests.Response):
try:
res.raise_for_status()
except requests.RequestException as e:
try:
msg = res.json()["msg"]
except: # noqa: E722
msg = ""
> raise PageserverApiException(msg, res.status_code) from e
E fixtures.pageserver.http.PageserverApiException: Layer index is corrupted
test_runner/fixtures/pageserver/http.py:125: PageserverApiException
--------------------------------
2023-06-09T13:30:55.115545Z INFO version: git:17bf24470db6ddb0b0ff334e6d8cfc968cfcf764-modified failpoints: true, features: ["testing"] launch_timestamp: 2023-06-09 13:30:55.111957976 UTC
2023-06-09T13:30:55.115734Z INFO Claimed pid file at "/tmp/test_output/test_pgbench[neon-45-400]/repo/pageserver.pid"
2023-06-09T13:30:55.115746Z INFO Starting pageserver http handler on localhost:15003
2023-06-09T13:30:55.116059Z INFO Starting pageserver pg protocol handler on localhost:15002
2023-06-09T13:30:55.117367Z INFO Using auth for http API: Trust
2023-06-09T13:30:55.117380Z INFO Using auth for pg connections: Trust
2023-06-09T13:30:55.117382Z INFO No JWT token for authentication with Safekeeper detected
2023-06-09T13:30:55.118298Z INFO init_tenant_mgr: Processed 0 local tenants at startup
2023-06-09T13:30:55.118358Z INFO Initial load completed elapsed_millis=0
2023-06-09T13:30:55.118380Z INFO Initial logical sizes completed from_init_done_millis=0 from_init_millis=0
2023-06-09T13:30:55.325596Z INFO request{method=POST path=/v1/tenant request_id=4b51754d-fa35-4d2c-9860-6fbc0a728fa0}: Handling request
2023-06-09T13:30:55.325732Z INFO request{method=POST path=/v1/tenant request_id=4b51754d-fa35-4d2c-9860-6fbc0a728fa0}:tenant_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf}:saving tenantconf: persisting tenantconf to /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/43e17795de1f42a27f5bfb2421a7a2cf.___temp/config
2023-06-09T13:30:55.325838Z INFO request{method=POST path=/v1/tenant request_id=4b51754d-fa35-4d2c-9860-6fbc0a728fa0}:tenant_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf}: tenant 43e17795de1f42a27f5bfb2421a7a2cf is assumed to be loadable, starting load operation
2023-06-09T13:30:55.325847Z INFO request{method=POST path=/v1/tenant request_id=4b51754d-fa35-4d2c-9860-6fbc0a728fa0}:tenant_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf}:spawn_load{tenant_id=43e17795de1f42a27f5bfb2421a7a2cf}: loading tenantconf from /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/43e17795de1f42a27f5bfb2421a7a2cf/config
2023-06-09T13:30:55.326101Z INFO load{tenant_id=43e17795de1f42a27f5bfb2421a7a2cf}: activation attempt finished since_creation_millis=0 tenant_id=43e17795de1f42a27f5bfb2421a7a2cf activated_timelines=0 total_timelines=0 post_state="Active"
2023-06-09T13:30:55.326201Z INFO compaction_loop{tenant_id=43e17795de1f42a27f5bfb2421a7a2cf}: starting
2023-06-09T13:30:55.326202Z INFO request{method=POST path=/v1/tenant request_id=4b51754d-fa35-4d2c-9860-6fbc0a728fa0}: Request handled, status: 201 Created
2023-06-09T13:30:55.326228Z INFO gc_loop{tenant_id=43e17795de1f42a27f5bfb2421a7a2cf}: starting
2023-06-09T13:30:55.326839Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}: Handling request
2023-06-09T13:30:55.327067Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: running /home/ars/neon/neon/pg_install/v14/bin/initdb in /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/43e17795de1f42a27f5bfb2421a7a2cf/timelines/basebackup-182fe66e445b124059b406e9f998187f.___temp, libdir: /home/ars/neon/neon/pg_install/v14/lib
2023-06-09T13:30:55.814489Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: importing slru file "pg_xact/0000"
2023-06-09T13:30:55.890888Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: importing slru file "pg_multixact/offsets/0000"
2023-06-09T13:30:55.890956Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: importing slru file "pg_multixact/members/0000"
2023-06-09T13:30:55.909284Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: reached end of WAL at 0/1696628
2023-06-09T13:30:55.909437Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: spawning flush loop
2023-06-09T13:30:55.909516Z INFO layer flush task{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline=182fe66e445b124059b406e9f998187f}: started flush loop
2023-06-09T13:30:55.925969Z INFO layer flush task{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline=182fe66e445b124059b406e9f998187f}:flush_frozen_layer{tenant_id=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f layer=inmem-00000000016965B0-0000000001696629}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:30:55.926605Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: skipping attempt to start flush_loop twice 43e17795de1f42a27f5bfb2421a7a2cf/182fe66e445b124059b406e9f998187f
2023-06-09T13:30:55.926615Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: created root timeline 182fe66e445b124059b406e9f998187f timeline.lsn 0/1696628
2023-06-09T13:30:55.933091Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=182fe66e445b124059b406e9f998187f lsn=None pg_version=Some(14)}: launching WAL receiver for timeline 182fe66e445b124059b406e9f998187f of tenant 43e17795de1f42a27f5bfb2421a7a2cf
2023-06-09T13:30:55.933160Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=0e78d915-5454-4d30-8972-838c81adfcb2}: Request handled, status: 201 Created
2023-06-09T13:30:55.936996Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}: Handling request
2023-06-09T13:30:55.937059Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=6acb521e53bd719f0ea8a1a16c37d043 lsn=None pg_version=None}: branching timeline 6acb521e53bd719f0ea8a1a16c37d043 from timeline 182fe66e445b124059b406e9f998187f at last record LSN: 0/1696628
2023-06-09T13:30:55.937215Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=6acb521e53bd719f0ea8a1a16c37d043 lsn=None pg_version=None}: loaded layer map with 0 layers at 0/1696628, total physical size: 0
2023-06-09T13:30:55.937229Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=6acb521e53bd719f0ea8a1a16c37d043 lsn=None pg_version=None}: spawning flush loop
2023-06-09T13:30:55.937246Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=6acb521e53bd719f0ea8a1a16c37d043 lsn=None pg_version=None}: branched timeline 6acb521e53bd719f0ea8a1a16c37d043 from 182fe66e445b124059b406e9f998187f at 0/1696628
2023-06-09T13:30:55.937250Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=6acb521e53bd719f0ea8a1a16c37d043 lsn=None pg_version=None}: launching WAL receiver for timeline 6acb521e53bd719f0ea8a1a16c37d043 of tenant 43e17795de1f42a27f5bfb2421a7a2cf
2023-06-09T13:30:55.937268Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=6acb521e53bd719f0ea8a1a16c37d043 lsn=None pg_version=None}: spawning logical size computation from context of task kind MgmtRequest
2023-06-09T13:30:55.937292Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}: Request handled, status: 201 Created
2023-06-09T13:30:55.937298Z INFO layer flush task{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline=6acb521e53bd719f0ea8a1a16c37d043}: started flush loop
2023-06-09T13:30:55.937377Z INFO request{method=POST path=/v1/tenant/43e17795de1f42a27f5bfb2421a7a2cf/timeline request_id=5358978d-d9a8-49ac-8d5a-86a37ad15a60}:timeline_create{tenant=43e17795de1f42a27f5bfb2421a7a2cf timeline_id=6acb521e53bd719f0ea8a1a16c37d043 lsn=None pg_version=None}:logical_size_calculation_task: Calculating logical size for timeline 6acb521e53bd719f0ea8a1a16c37d043 at 0/1696628
2023-06-09T13:30:55.941063Z INFO request{method=POST path=/v1/tenant request_id=4e8613a1-1a17-43c5-8a60-2242b3fc9580}: Handling request
2023-06-09T13:30:55.941146Z INFO request{method=POST path=/v1/tenant request_id=4e8613a1-1a17-43c5-8a60-2242b3fc9580}:tenant_create{tenant=fa34cc085bb558135b4c8dc5a58db944}:saving tenantconf: persisting tenantconf to /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944.___temp/config
2023-06-09T13:30:55.941215Z INFO request{method=POST path=/v1/tenant request_id=4e8613a1-1a17-43c5-8a60-2242b3fc9580}:tenant_create{tenant=fa34cc085bb558135b4c8dc5a58db944}: tenant fa34cc085bb558135b4c8dc5a58db944 is assumed to be loadable, starting load operation
2023-06-09T13:30:55.941223Z INFO request{method=POST path=/v1/tenant request_id=4e8613a1-1a17-43c5-8a60-2242b3fc9580}:tenant_create{tenant=fa34cc085bb558135b4c8dc5a58db944}:spawn_load{tenant_id=fa34cc085bb558135b4c8dc5a58db944}: loading tenantconf from /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/config
2023-06-09T13:30:55.941425Z INFO load{tenant_id=fa34cc085bb558135b4c8dc5a58db944}: activation attempt finished since_creation_millis=0 tenant_id=fa34cc085bb558135b4c8dc5a58db944 activated_timelines=0 total_timelines=0 post_state="Active"
2023-06-09T13:30:55.941462Z INFO request{method=POST path=/v1/tenant request_id=4e8613a1-1a17-43c5-8a60-2242b3fc9580}: Request handled, status: 201 Created
2023-06-09T13:30:55.941482Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}: starting
2023-06-09T13:30:55.941504Z INFO gc_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}: starting
2023-06-09T13:30:55.941765Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}: Handling request
2023-06-09T13:30:55.941855Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: running /home/ars/neon/neon/pg_install/v14/bin/initdb in /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/basebackup-c534ad212640ab5295bcf406072364cd.___temp, libdir: /home/ars/neon/neon/pg_install/v14/lib
2023-06-09T13:30:56.338176Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: importing slru file "pg_xact/0000"
2023-06-09T13:30:56.419258Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: importing slru file "pg_multixact/offsets/0000"
2023-06-09T13:30:56.419336Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: importing slru file "pg_multixact/members/0000"
2023-06-09T13:30:56.436991Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: reached end of WAL at 0/1696628
2023-06-09T13:30:56.437011Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: spawning flush loop
2023-06-09T13:30:56.437091Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=c534ad212640ab5295bcf406072364cd}: started flush loop
2023-06-09T13:30:56.452979Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=c534ad212640ab5295bcf406072364cd}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd layer=inmem-00000000016965B0-0000000001696629}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:30:56.454067Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: skipping attempt to start flush_loop twice fa34cc085bb558135b4c8dc5a58db944/c534ad212640ab5295bcf406072364cd
2023-06-09T13:30:56.454077Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: created root timeline c534ad212640ab5295bcf406072364cd timeline.lsn 0/1696628
2023-06-09T13:30:56.460051Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=c534ad212640ab5295bcf406072364cd lsn=None pg_version=Some(14)}: launching WAL receiver for timeline c534ad212640ab5295bcf406072364cd of tenant fa34cc085bb558135b4c8dc5a58db944
2023-06-09T13:30:56.460110Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=ad032cbf-0ebd-4e84-83db-c8716a0734f8}: Request handled, status: 201 Created
2023-06-09T13:30:56.463773Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}: Handling request
2023-06-09T13:30:56.463901Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: running /home/ars/neon/neon/pg_install/v14/bin/initdb in /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/basebackup-1360a54b2bd92aa3e9d6c90d3404a976.___temp, libdir: /home/ars/neon/neon/pg_install/v14/lib
2023-06-09T13:30:56.845298Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: importing slru file "pg_xact/0000"
2023-06-09T13:30:56.916391Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: importing slru file "pg_multixact/offsets/0000"
2023-06-09T13:30:56.916454Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: importing slru file "pg_multixact/members/0000"
2023-06-09T13:30:56.933634Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: reached end of WAL at 0/1696628
2023-06-09T13:30:56.933844Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: spawning flush loop
2023-06-09T13:30:56.933932Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}: started flush loop
2023-06-09T13:30:56.950975Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000016965B0-0000000001696629}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:30:56.951918Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: skipping attempt to start flush_loop twice fa34cc085bb558135b4c8dc5a58db944/1360a54b2bd92aa3e9d6c90d3404a976
2023-06-09T13:30:56.951925Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: created root timeline 1360a54b2bd92aa3e9d6c90d3404a976 timeline.lsn 0/1696628
2023-06-09T13:30:56.957556Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}:timeline_create{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None pg_version=Some(14)}: launching WAL receiver for timeline 1360a54b2bd92aa3e9d6c90d3404a976 of tenant fa34cc085bb558135b4c8dc5a58db944
2023-06-09T13:30:56.957612Z INFO request{method=POST path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline request_id=1ef74915-2176-4044-b8d8-d9d5b88c76d4}: Request handled, status: 201 Created
2023-06-09T13:30:56.980425Z INFO handle_basebackup_request{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None prev_lsn=None full_backup=false}: taking basebackup lsn=0/1696628, prev_lsn=0/16965B0 (full_backup=false)
2023-06-09T13:30:56.980872Z INFO handle_basebackup_request{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None prev_lsn=None full_backup=false}:send_tarball{backup_lsn=0/1696628}: timeline.pg_version 14
2023-06-09T13:30:56.989608Z INFO handle_basebackup_request{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 lsn=None prev_lsn=None full_backup=false}: basebackup complete lsn_await_millis=0 basebackup_millis=9
2023-06-09T13:30:57.227848Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}: New SK node was added: 1
2023-06-09T13:30:57.227893Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}: Switching to new connection candidate: NewWalConnectionCandidate { safekeeper_id: NodeId(1), wal_source_connconf: PgConnectionConfig { host: Ipv4(127.0.0.1), port: 15004, password: None }, availability_zone: Some("sk-1"), reason: NoExistingConnection }
2023-06-09T13:30:57.227942Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: connecting to PgConnectionConfig { host: Ipv4(127.0.0.1), port: 15004, password: None }
2023-06-09T13:30:57.228434Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: IdentifySystem { systemid: 7242678326296251714, timeline: 1, xlogpos: 0/17348A8, dbname: None }
2023-06-09T13:30:57.228445Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: last_record_lsn 0/1696628 starting replication from 0/1696628, safekeeper is at 0/17348A8...
2023-06-09T13:30:57.234246Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: caught up at LSN 0/17348A8
2023-06-09T13:31:01.725665Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243046783, layer size 268501388, elapsed since last flush 4.880945917s
2023-06-09T13:31:02.799141Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000001696629-000000000FE5FFA9}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:03.966849Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007520, layer size 268444602, elapsed since last flush 2.236550503s
2023-06-09T13:31:04.979002Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-000000000FE5FFA9-000000001E61FFC9}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:06.298982Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007504, layer size 268444602, elapsed since last flush 2.327841178s
2023-06-09T13:31:07.420629Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-000000001E61FFC9-000000002CDDFFD9}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:08.894090Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007520, layer size 268444602, elapsed since last flush 2.589368458s
2023-06-09T13:31:09.990662Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-000000002CDDFFD9-000000003B59FFF9}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:11.477792Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007464, layer size 268442959, elapsed since last flush 2.578339335s
2023-06-09T13:31:12.594763Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-000000003B59FFF9-0000000049D5FFE1}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:14.091431Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007520, layer size 268444602, elapsed since last flush 2.609141071s
2023-06-09T13:31:15.209510Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000049D5FFE1-0000000058520001}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:16.599558Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007424, layer size 268444026, elapsed since last flush 2.503511703s
2023-06-09T13:31:17.769269Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000058520001-0000000066CDFFC1}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:19.230607Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007520, layer size 268444602, elapsed since last flush 2.626231477s
2023-06-09T13:31:20.423212Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000066CDFFC1-000000007549FFE1}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:21.780699Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007416, layer size 268444071, elapsed since last flush 2.545260265s
2023-06-09T13:31:22.875818Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-000000007549FFE1-0000000083C5FF99}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:24.491882Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007560, layer size 268444470, elapsed since last flush 2.705807089s
2023-06-09T13:31:25.702622Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000083C5FF99-000000009241FFE1}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:27.228578Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007448, layer size 268441774, elapsed since last flush 2.731759463s
2023-06-09T13:31:28.299460Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-000000009241FFE1-00000000A0BDFFB9}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:29.731196Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007456, layer size 268444248, elapsed since last flush 2.495994831s
2023-06-09T13:31:30.849612Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000A0BDFFB9-00000000AF39FF99}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:32.336024Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007504, layer size 268444602, elapsed since last flush 2.600350258s
2023-06-09T13:31:33.463511Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000AF39FF99-00000000BDB5FFA9}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:34.809747Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007440, layer size 268443465, elapsed since last flush 2.469191788s
2023-06-09T13:31:35.870216Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000BDB5FFA9-00000000CC31FF79}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:36.680532Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: Starting Level0 compaction in LSN range 0/16965B0-0/CC31FF79 for 15 layers (15 deltas in total)
2023-06-09T13:31:36.680556Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000016965B0-0000000001696629
2023-06-09T13:31:36.680561Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000001696629-000000000FE5FFA9
2023-06-09T13:31:36.680565Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000000FE5FFA9-000000001E61FFC9
2023-06-09T13:31:36.680569Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000001E61FFC9-000000002CDDFFD9
2023-06-09T13:31:36.680577Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000002CDDFFD9-000000003B59FFF9
2023-06-09T13:31:36.680581Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000003B59FFF9-0000000049D5FFE1
2023-06-09T13:31:36.680585Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000049D5FFE1-0000000058520001
2023-06-09T13:31:36.680588Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000058520001-0000000066CDFFC1
2023-06-09T13:31:36.680592Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000066CDFFC1-000000007549FFE1
2023-06-09T13:31:36.680595Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000007549FFE1-0000000083C5FF99
2023-06-09T13:31:36.680599Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000083C5FF99-000000009241FFE1
2023-06-09T13:31:36.680602Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000009241FFE1-00000000A0BDFFB9
2023-06-09T13:31:36.680606Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000A0BDFFB9-00000000AF39FF99
2023-06-09T13:31:36.680609Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000AF39FF99-00000000BDB5FFA9
2023-06-09T13:31:36.680613Z INFO compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}:blocking: compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000BDB5FFA9-00000000CC31FF79
2023-06-09T13:31:37.392465Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007504, layer size 268444602, elapsed since last flush 2.577673744s
2023-06-09T13:31:39.057129Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000CC31FF79-00000000DAADFF89}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:39.903838Z INFO request{method=PUT path=/v1/tenant/fa34cc085bb558135b4c8dc5a58db944/timeline/1360a54b2bd92aa3e9d6c90d3404a976/checkpoint request_id=870abf0c-dae2-4c2f-91bf-ec7990b9d783}: Handling request
2023-06-09T13:31:40.589777Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000DAADFF89-00000000E3A8EFD9}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:42.000991Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243077007, layer size 268520780, elapsed since last flush 4.602891038s
2023-06-09T13:31:42.946802Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000E3A8EFD9-00000000F225FF69}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:44.094103Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007504, layer size 268444602, elapsed since last flush 2.088805709s
2023-06-09T13:31:45.073664Z INFO layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-00000000F225FF69-0000000100A1FF79}:blocking: Rebuilt layer map. Did 1 insertions to process a batch of 1 updates.
2023-06-09T13:31:46.188981Z INFO wal_connection_manager{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976}:connection{node_id=1}: check_checkpoint_distance 243007584, layer size 268443558, elapsed since last flush 2.090442061s
2023-06-09T13:31:47.135287Z ERROR layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000100A1FF79-000000010F1DFFD9}:blocking: writeback of buffer EphemeralPage { file_id: 23, blkno: 13136 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:47.135383Z ERROR layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000100A1FF79-000000010F1DFFD9}:blocking: writeback of buffer EphemeralPage { file_id: 23, blkno: 13137 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:47.135450Z ERROR layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000100A1FF79-000000010F1DFFD9}:blocking: writeback of buffer EphemeralPage { file_id: 23, blkno: 13138 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:47.135537Z ERROR layer flush task{tenant=fa34cc085bb558135b4c8dc5a58db944 timeline=1360a54b2bd92aa3e9d6c90d3404a976}:flush_frozen_layer{tenant_id=fa34cc085bb558135b4c8dc5a58db944 timeline_id=1360a54b2bd92aa3e9d6c90d3404a976 layer=inmem-0000000100A1FF79-000000010F1DFFD9}:block2023-06-09T13:31:53.756991Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 14424 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757002Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 15788 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757009Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 21099 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757016Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 17048 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757022Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 18625 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757028Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 18626 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757034Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 18162 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757041Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 20928 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.757047Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 14425 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No 2023-06-09T13:31:53.760627Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 20454 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760640Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 15885 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760645Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 14798 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760649Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 20945 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760653Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 14528 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760656Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 18223 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760660Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 19093 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760664Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 21145 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No space left on device (os error 28)
2023-06-09T13:31:53.760668Z ERROR compaction_loop{tenant_id=fa34cc085bb558135b4c8dc5a58db944}:compact_timeline{timeline=1360a54b2bd92aa3e9d6c90d3404a976}: writeback of buffer EphemeralPage { file_id: 23, blkno: 17395 } failed: failed to write back to ephemeral file at /tmp/test_output/test_pgbench[neon-45-400]/repo/tenants/fa34cc085bb558135b4c8dc5a58db944/timelines/1360a54b2bd92aa3e9d6c90d3404a976/ephemeral-23 error: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment