Opened 4 years ago

Last modified 4 years ago

#2902 new defect

EC2 spot instances fail on InvalidSpotInstanceRequestID.NotFound

Reported by: bgilbert Owned by:
Priority: major Milestone: undecided
Version: 0.8.9 Keywords: ec2
Cc:

Description

After EC2LatentBuildSlave creates a spot request, it immediately queries it using get_all_spot_instance_requests(). Since the EC2 API uses eventual consistency, the query can fail (and does, fairly reliably) with InvalidSpotInstanceRequestID.NotFound. EC2LatentBuildSlave treats this as a fatal error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 525, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
  File "/home/foo/env/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/home/foo/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/foo/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/foo/env/local/lib/python2.7/site-packages/buildbot/buildslave/ec2.py", line 362, in _request_spot_instance
    request = self._wait_for_request(reservations[0])
  File "/home/foo/env/local/lib/python2.7/site-packages/buildbot/buildslave/ec2.py", line 406, in _wait_for_request
    request_ids=[reservation.id])
  File "/home/foo/env/local/lib/python2.7/site-packages/boto/ec2/connection.py", line 1342, in get_all_spot_instance_requests
    [('item', SpotInstanceRequest)], verb='POST')
  File "/home/foo/env/local/lib/python2.7/site-packages/boto/connection.py", line 1166, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSpotInstanceRequestID.NotFound</Code><Message>The spot instance request ID 'sir-0231f26g' does not exist</Message></Error></Errors><RequestID>cade547a-bf90-482f-b44b-2a7f5b3f8c01</RequestID></Response>

Change History (1)

comment:1 Changed 4 years ago by sa2ajj

  • Keywords ec2 added
Note: See TracTickets for help on using tickets.