java - Why do more requests go to new (dynamic) instances than to resident instance?

Question

Have a Java app with auto-scaling on App Engine Standard Environment. Right now the scaling is configured like this:

<instance-class>F2</instance-class>

<automatic-scaling>
    <min-idle-instances>1</min-idle-instances>

    <!-- ‘automatic’ is the default value. -->
    <max-idle-instances>2</max-idle-instances>

    <!-- ‘automatic’ is the default value. -->
    <min-pending-latency>2000ms</min-pending-latency>

    <max-pending-latency>8000ms</max-pending-latency>
    <max-concurrent-requests>60</max-concurrent-requests>
</automatic-scaling>

Just started trying the F2 instance, was using F1 instances earlier. No matter how I configure my auto-scaling, it seems like the newly-created instance (created when load increases) starts getting all the incoming requests, while the resident instance sits with a very light load.

Why is this? Of course, I am unable to monitor the traffic (and which instance it goes to) in real time, but every time I look the story seems to be the same. I have included a few sample screenshots below.

and in the following case three instances (this was a slightly different configuration from the one above) are sitting free, but GAE's load balancer chooses to send all requests to the instance with the highest latency!

One more example: this is the request log for the resident instance started at 10:15:45 AM today:

and the request log for the dynamic instance that started 10 seconds later:

As you can see the dynamic instance is handling all of the requests (1889 so far) while the resident sits essentially idle (7 in the same time period). This would still be OK if not for the fact that the resident instances seem to be destroyed and created anew right around the time new dynamic instances are being created. This means that for a minute or so all requests see 10-20 second response times.

Can somebody please explain to me how to configure?

Here's what I want:

• One idle instance should be able to handle the load most of the times (for now).
• When more requests come in, spin up an additional instance. When it is ready, start diverting traffic to it.

I am trying to run a reasonable-load site on a shoestring budget, so it is important that I try to stay as close to the free quota as possible.

Update 1

Since both the answers talk about the warmup request prominently, I thought I'd list details about it here. I am using a ServletContextListener to handle the initialization. It does the following (times are gathered using Guava's Stopwatch class and are for the code I have written/am explicitly invoking):

1. Register Objectify entities (1.449 s)
2. Freemarker init 229 ms
3. Firebase init 228.2 ms

Other than that I have the Shiro filter, the Objectify filter and Jersey filter (in Jersey I am avoiding classpath scanning (I think) by explicitly registering the classes rather than giving it a package to scan) configured in my web.xml. Not using any dependency injection to avoid classpath scanning.

The /_ah/warmup request took 7.8s (the one from where the above times are taken). But requests being served by a freshly-started dynamic instance whose warmup has already finished are taking 10+ seconds to complete, despite the fact that these same calls take 200-700ms two minutes later. So what else is going on in the background other than the stuff I am explicitly doing in my StartupListener?

Here's part 1 of the log and here's part 2 of the log.

Answer 1

Waitting for answers