python - Ray: autoscaler on aws only creates head node, not worker nodes?

Question

This is my yaml file:

cluster_name: my_test

# The number of worker nodes to launch in addition to the head node.
initial_workers: 3
min_workers: 3
max_workers: 3

provider:
 type: aws
 region: us-east-2
 availability_zone: us-east-2a

head_node:
    InstanceType: r5.4xlarge
    ImageId: ami-0ebc84cad4ab1c308 # Ubuntu 18.04
    BlockDeviceMappings:
        - DeviceName: /dev/sda1
          Ebs:
              VolumeSize: 10

    InstanceMarketOptions:
        MarketType: spot
        SpotOptions:
            MaxPrice: '0.18'


worker_nodes:
    InstanceType: r5.4xlarge
    ImageId: ami-0ebc84cad4ab1c308 # Ubuntu 18.04

    BlockDeviceMappings:
        - DeviceName: /dev/sda1
          Ebs:
              VolumeSize: 10

    InstanceMarketOptions:
        MarketType: spot
        SpotOptions:
            MaxPrice: '0.18'

# List of shell commands to run to set up nodes.
setup_commands:
    - sudo apt update && sudo apt install -y gcc unzip libc6-dev g++
    - wget https://repo.continuum.io/miniconda/Miniconda3-py38_4.8.3-Linux-x86_64.sh || true
    - bash Miniconda3-py38_4.8.3-Linux-x86_64.sh -b -p $HOME/miniconda3 || true
    - echo 'export PATH="$HOME/miniconda3/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc      
    - source ~/miniconda3/etc/profile.d/conda.sh && pip install ray[tune]==1.0.1.post1

When I run ray up my_file.yaml, it only creates the head node without any worker nodes.

But when I change InstanceType to something else, say m4.xlarge, or change region to west, it creates worker nodes. However, If I tear down all and up ray again, no workers are made again. The Worst is that this happened randomly!

What's wrong with it?

question from:https://stackoverflow.com/questions/65541034/ray-autoscaler-on-aws-only-creates-head-node-not-worker-nodes

Answer 1

Waitting for answers