Skip to content

Instantly share code, notes, and snippets.

@bzp2010

bzp2010/APISIX-plugin-multi-dc.md

Last active October 13, 2023 09:45
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bzp2010/109e51f1addf7050bb61a54546c478dd to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bzp2010/109e51f1addf7050bb61a54546c478dd to your computer and use it in GitHub Desktop.
Download ZIP
APISIX cross-datacenter call plugin (dynamic upstream)
Raw
APISIX-plugin-multi-dc.md

Apache APISIX's external service call multi-dc plugin demo

Logic

We define a configuration group in the plugin configuration that includes each data center and the services within it. It allows to configure a separate upstream for each service, while using the datacenter-level upstream as a fallback.

When the request arrives, we parse the content of the source Host and extract the target data center and services from it.

Then it will read that desired upstream from our predefined list of upstreams and set it as the upstream for this request. In this process, the request header and request body will be sent directly to the target service without being modified.

Usage

  1. Install the plugin to APISIX and add it to the plugin list.

  2. Create some upstreams for testing

## data center level
curl -XPUT 'http://127.0.0.1:9080/apisix/admin/upstreams/test_dc1' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "nodes": {
        "httpbin.org:80": 1
    },
    "type": "roundrobin"
}'
## service level #1
curl -XPUT 'http://127.0.0.1:9080/apisix/admin/upstreams/test_dc1_service1' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "nodes": {
        "httpbin.org:80": 1
    },
    "type": "roundrobin"
}'
## service level #2
curl -XPUT 'http://127.0.0.1:9080/apisix/admin/upstreams/test_dc1_service2' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "nodes": {
        "httpbin.org:80": 1
    },
    "type": "roundrobin"
}'

All the upstream nodes above are configured as httpbin.org, for your testing you can replace them with other services.

  1. Create route for testing

You can see that we configured a data center with one services, and data center and service is configured with upstream. One of the upstream test_dc1_service2 is not configured as a service object, so it will fallback to the data center level.

Tip: In order to avoid nested upstreams that lead to overly complex configuration structures, this plugin only allows the use of upstream_id.

curl -XPUT 'http://127.0.0.1:9080/apisix/admin/routes/1' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "uri": "/*",
    "plugins": {
        "multi-dc": {
            "data_centers": {
                "destination-datacenter1": {
                    "services": {
                        "example-application1": {
                            "upstream_id": "test_dc1_service1"
                        }
                    },
                    "upstream_id": "test_dc1"
                }
            }
        }
    }
}'
  1. let's try it
curl -XGET 'http://127.0.0.1:9080/get' \
--header 'Host: example-application.destination-datacenter1-dci.source-datacenter.example.com'

Note that host, the host above was originally example-application-destination-datacenter1-dci, but we didn't actually have the ability to perform a regular expression on this to extract the service name and data center name, so I modified it to the above format with the service name and data center name using . split.

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "example-application.destination-datacenter1-dci.source-datacenter.example.com",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-6332c583-70c3c9c0446464c00a25c91c",
    "X-Forwarded-Host": "example-application.destination-datacenter1-dci.source-datacenter.example.com"
  },
  "origin": "127.0.0.1",
  "url": "http://example-application.destination-datacenter1-dci.source-datacenter.example.com/get"
}
Raw
multi-dc.lua
local core = require("apisix.core")
local upstream = require("apisix.upstream")
local re_sub = ngx.re.sub
local re_gmatch = ngx.re.gmatch
local UPSTREAM_SOURCE = {
SERVICE = "service",
DATA_CENTER = "data_center"
}
local id_schema = {
anyOf = {
{
type = "string", minLength = 1, maxLength = 64,
pattern = [[^[a-zA-Z0-9-_.]+$]]
},
{ type = "integer", minimum = 1 }
}
}
local services_schema = {
type = "object",
patternProperties = {
[".*"] = {
description = "service items",
type = "object",
properties = {
upstream_id = id_schema,
},
required = {"upstream_id"},
minimum = 1,
},
},
additionalProperties = false,
}
local data_centers_schema = {
type = "object",
patternProperties = {
[".*"] = {
description = "data center items",
type = "object",
properties = {
services = services_schema,
upstream_id = id_schema,
},
anyOf = {
{required = {"services", "upstream_id"}},
{required = {"services"}},
{required = {"upstream_id"}}
},
minimum = 1,
},
},
additionalProperties = false,
}
local schema = {
type = "object",
properties = {
data_centers = data_centers_schema,
},
required = {"data_centers"}
}
local _M = {
version = 0.1,
priority = 967,
name = "multi-dc",
schema = schema,
}
function _M.check_schema(conf)
return core.schema.check(schema, conf)
end
local function set_upstream(upstream_info, ctx)
local up_conf = {
name = upstream_info.name,
type = upstream_info.type,
hash_on = upstream_info.hash_on,
pass_host = upstream_info.pass_host,
upstream_host = upstream_info.upstream_host,
key = upstream_info.key,
nodes = upstream_info.nodes,
timeout = upstream_info.timeout,
}
local ok, err = upstream.check_schema(up_conf)
if not ok then
core.log.error("failed to validate generated upstream: ", err)
return 500, err
end
local matched_route = ctx.matched_route
up_conf.parent = matched_route
local upstream_key = up_conf.type .. "#route_" ..
matched_route.value.id .. "_multi-dc_" .. upstream_info.id
core.log.info("upstream_key: ", upstream_key)
upstream.set(ctx, upstream_key, ctx.conf_version, up_conf)
return
end
function _M.access(conf, ctx)
-- extract target data_center and service
local segment = {}
for str in re_gmatch(ctx.var.host, "([^.]+)") do
table.insert(segment, str[1])
end
-- check target
if not segment[1] or not segment[2] then
return 503, {error_msg = "incorrect target syntax"}
end
-- remove "-dci" suffix
segment[2] = re_sub(segment[2], "^(.*)-dci", "$1")
core.log.debug("try to access external service, data center: ", segment[2],
", service: " .. segment[1])
-- find target upstream
local target_upstream_id, target_upstream_source
local target_data_center = conf.data_centers[segment[2]]
if target_data_center ~= nil then
if target_data_center.services ~= nil and
target_data_center.services[segment[1]] ~= nil then
target_upstream_id = target_data_center.services[segment[1]].upstream_id
target_upstream_source = UPSTREAM_SOURCE.SERVICE
end
if target_upstream_id == nil and target_data_center.upstream_id ~= nil then
target_upstream_id = target_data_center.upstream_id
target_upstream_source = UPSTREAM_SOURCE.DATA_CENTER
end
end
if not target_upstream_id then
core.log.warn("cannot find target upstream")
return 503, {error_msg = "no target upstream"}
end
-- dynamic set upstream
core.log.debug("selected upstream id: ", target_upstream_id, ", source: ", target_upstream_source)
local target_upstream = upstream.get_by_id(target_upstream_id)
core.log.debug("upstream info: ", target_upstream)
if not target_upstream then
core.log.warn("target upstream not exist: ", target_upstream_id)
return 503, {error_msg = "target upstream not exist"}
end
-- override upstream host
ctx.var.upstream_host = segment[1] .. "." .. segment[2] .. ".trendyol.com"
return set_upstream(target_upstream, ctx)
end
return _M
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment