New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bulk delete can appear to timeout #7732
Comments
I would have assumed that the implementation returns immediately and the delete is eventually consistent. |
I am using the Grizzly (2013.1) version of Openstack Swift. I think the bulk middleware has been changed since then and it may well return immediately in the Havana version of Openstack swift. I will do some further research - thanks! |
I have tested a bulk delete with Rackspace US this morning. The results were not positive. I created a container called bulk-test with 2000 200k files in it. I then deleted the container in Cyberduck 4.3.2. My timeout was set to 30secs The connection did NOT timeout, as I see with my Openstack swift cluster. After the DELETE request was made, the server responded 200 OK immediately:
However, the GUI did not return to a good state after the 200 OK message I could not issue further commands (eg. refresh on the container list). See the attached screenshot. The GUI remained in the state in the screenshot for 11mins, then I saw a popup error and the following log message:
Oddly, I then refreshed the container list and opened bulk-test - it showed four objects. That means 1996 files were removed correctly. The full logs file is below:
|
The response of the bulk delete request has a |
We have tweaked some setting in the Havana release of Openstack switt (namely lowering the default yield_frequency variable of the bulk middleware from 60 to 10) and we think this is now working. We're still confirming, but it looks like Cyberduck is blameless. David. |
|
I suppose this is fixed with Swift release 1.12.0
|
The recent fix to the yield_frequency in combination with the older bulk middleware refactoring (openstack/swift@af2607c ) should give us more reliable bulk deletions. |
As I understand it, the bulk-delete multi object deletion in openstack swift takes a list of objects and does not return until the objects have been removed (assuming success). I don't believe there's any network I/O while the bulk-delete is working, so if the operation takes more than the connection timeout value ( preference connection.timeout.seconds) then the connection is closed locally. I had the default 30second timeout set when I ran into this:
Here's the log messages I saw when deleting 500 objects:
Notice it's 30seconds from :
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >> Accept-Encoding: gzip,deflate
to
2014-01-11 14:40:30,711 [background-1] DEBUG org.apache.http.impl.conn.DefaultClientConnection - Connection 0.0.0.0:56108<->193.203.66.227:443 closed
The 5 retries did not help in this case. It's properly a bad idea to re-send a duplicate bulk-delete request several times...
Setting connection.timeout.seconds to 600 seconds helped me to delete my large data set. Would love to get a fix for this so we can delete 50000+ objects and still retain the default 30 sec connection timeout
Attachments
Screen Shot 2014-01-13 at 11.42.55.png
(146.9 KiB)The text was updated successfully, but these errors were encountered: