Why these two can be regional/edge-optimized separately?
Regional and Edge-Optimized are deployment options. Neither option changes anything fundamental about how the API is processed by the AWS infrastructure once the request arrives at the core of the API Gateway service or how the services behind API Gateway ultimately are accessed -- what changes is how the requests initially arrive at AWS and are delivered to the API Gateway core for execution. More about this, below.
When you use a custom domain name, your selected API stage is deployed a second time, on a second endpoint, which is why you have a second selection of a deployment type that must be made.
Each endpoint has the characteristics of its deployment type, whether regional or edge-optimized. The original deployment type of the API itself does not impact the behavior of the API if deployed with a custom domain name, and subsequently accessed using that custom domain name -- they're independent.
Typically, if you deploy your API with a custom domain name, you wouldn't continue to use the deployment endpoint created for the main API (e.g. xxxx.execute-api.{region}.amazonaws.com), so the initial selection should not matter.
And when do I want each of them to be regional/edge-optimized?
If you're using a custom domain name, then, as noted above, your original deployment selection for the API as a whole has no further impact when you use the custom domain.
Edge-optimized endpoints were originally the only option available. If you don't have anything on which to base your selection, this choice is usually a reasonable one.
This option routes incoming requests through the AWS "Edge Network," which is the CloudFront network, with its 100+ global edge locations. This does not change where the API Gateway core ultimately handles your requests -- they are still ultimately handled within the same region -- but the requests are routed from all over the world into the nearest AWS edge, and they travel from there on networks operated by AWS to arrive at the region where you deployed your API.
If the clients of your API Gateway stage are globally dispersed, and you are only deploying your API in a single region, you probably want an edge-optimized deployment.
The edge-optimized configuration tends to give you better global responsiveness, since it tends to reduce the impact of network round trips, and the quality of the transport is not subject to as many of the vagaries of the public Internet because the request covers the least amount of distance possible before jumping off the Internet and onto the AWS network. The TCP handshake and TLS are negotiated with the connecting browser/client across a short distance (from the client to the edge) and the edge network maintains keep-alive connections that can be reused, all of which usually works in your favor... but this optimization becomes a relative impairment when your clients are always (or usually) calling the API from within the AWS infrastructure, within the same region, since the requests need to hop over to the edge network and then back into the core regional network.
If the clients of your API Gateway stage are inside AWS and within the same region where you deployed the API (such as when the API is being called by other systems in EC2 within the region), then you will most likely want a regional endpoint. Regional endpoints route requests through less of the AWS infrastructure, ensuring minimal latency and reduced jitter when requests are coming from EC2 within the same region.
As a side-effect of routing through the edge network, edge-optimized endpoints also provide some additional request headers that you may find useful, such as CloudFront-Viewer-Country: XX which attempts to identify the two-digit country code of the geographic location of the client making the API request. Regional endpoints don't have these headers.
As a general rule, go with edge-optimized unless you find a reason not to.
What would be some reasons not to? As mentioned above, if you or others are calling the API from within the same AWS region, you probably want a regional endpoint. Edge-optimized endpoints can introduce some edge-case side-effects in more advanced or complicated configurations, because of the way they integrate into the rest of the infrastructure. There are some things you can't do with an edge-optimized deployment, or that are not optimal if you do:
if you are using CloudFront for other sites, unrelated to API Gateway, and CloudFront is configured for a wildcard alternate domain name, like *.example.com, then you can't use a subdomain from that wildcard domain, such as api.example.com, on an edge-optimized endpoint with a custom domain name, because API Gateway submits a request to the edge network on your behalf to claim all requests for that subdomain when they arrive via CloudFront, and CloudFront rejects this request since it represents an unsupported configuration when used with API Gateway, even though CloudFront supports it in some other circumstances.
if you want to provide redundant APIs that respond to the same custom domain name in multiple regions, and use Route 53 Latency-Based Routing to deliver requests to the region nearest to the requester, you can't do this with an edge-optimized custom domain, because the second API Gateway region will not be able to claim the traffic for that subdomain on the edge network, since the edge network requires exactly 1 target for any given domain name (subdomain). This configuration can be achieved using regional endpoints and Route 53 LBR, or can be achieved while leveraging the edge network by using your own CloudFront distribution, Lambda@Edge to select the target endpoint based on the caller's location, and API Gateway regional deployments. Note that this can't be achieved by any means if you need to support IAM authentication of the caller, because the caller needs to know the target region before signing and submitting the request.
if you want to use your API as part of a larger site that integrates multiple resources, is deployed behind CloudFront, and uses the path to route to different services -- for example, /images/* might route to an S3 bucket, /api/* might route to your API Gateway stage, and * (everything else) might route to an elastic load balancer in EC2 -- then you don't want to use an edge-optimized API, because this causes your requests to loop through the edge network twice (increasing latency) and causes some header values to be lost. This configuration doesn't break, but it isn't optimal. For this, a regional endpoint is desirable.
