Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.CDN_ACCESS_LOGS Folder listing is empty #5350

Closed
cyberduck opened this issue Oct 20, 2010 · 14 comments
Closed

.CDN_ACCESS_LOGS Folder listing is empty #5350

cyberduck opened this issue Oct 20, 2010 · 14 comments
Assignees
Labels
bug cloudfiles Rackspace CloudFiles fixed
Milestone

Comments

@cyberduck
Copy link
Collaborator

d72cbab created the issue

CloudFiles Web interface shows heaps of files in this folder. I have successfully downloaded them using the cloudfiles-python module ... but CyberDuck shows this folder as being empty, despite the HTTP transcript saying there are 695 objects in the folder.
Screenshots attached.

FWIW .. here is some ropey python code that downloads the contents of CDN_ACCESS_LOGS ..

#mainline
ofolder = ".CDN_ACCESS_LOGS"
try:
    conn = cloudfiles.get_connection(username, apikey)
    container = conn.get_container(ofolder)
    lst = container.list_objects()
    for x in lst:
        obj = container.get_object(x)
        bn = os.path.split(x)
        savename = './cdnlogs/'+bn[1]
        if os.path.exists(savename):
            continue
        obj.save_to_filename(savename)
        print bn[1]+'\n'
except cloudfiles.errors.ResponseError, err:
    print ifile, ofolder, err

Attachments

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I cannot replicate the issue here using version 3.7. Can you verify this is still not working for you with 3.7?

@cyberduck
Copy link
Collaborator Author

d72cbab commented

Just downloaded 3.7
Afraid it still shows blank listing.
Screenshot attached.

Thanks,
Dave

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The only thing is that the Content-Length returned is suspiciously short. I doubt it can contain a XML listing of the advertised 697 objects.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Can you please contact Rackspace support and provide them the transcript from the log drawer in Cyberduck.

@cyberduck
Copy link
Collaborator Author

http://openid-provider.appspot.com/shawncmaddock commented

I've been having the identical issue. (Other containers work correctly, .CDN_ACCESS_LOGS shows contents in the web interface but not Cyberduck, small content length...) I contacted Rackspace, and included my Cyberduck log. Here is their response:

The problem with CyberDuck and other third-party applications such as FireUploader when it comes to the CDN log container is that they each have their own unique implementations of folder structures within Cloud Files. CDN access logs are stored in the format "containername/yyyy/mm/dd/hh/uniqueid.log.0.gz". The slashes in the object name are interpreted as subdirectories by CyberDuck, but since the corresponding "folder objects" are not there (in this case, it would require objects named containername, containername/yyyy, containername/yyyy/mm, etc.), it cannot read the objects in the container properly.

The only options for fixing this problem is if Cyberduck either:

A. automatically adds the corresponding folder objects inside .CDN_ACCESS_LOGS for you
B. updates their code to ignore the slashes in the .CDN_ACCESS_LOGS container entirely

Option A is a bad idea in general because it would make it quite difficult to find the log you're looking for and it would conflict with every other implementation. FireUploader requires a slightly different MIME type and metadata in their folder objects, for example. Option B is far more sensible.

Unfortunately, there's nothing that we can do on our end to accommodate the idiosyncrasies of all the third-party folder implementations.

Seeing as the specs for Cloud Files do not allow for nested containers, couldn't the "exception" be applied to all Cloud Files containers, and treat the / as a "normal" character? OS X doesn't have a problem with it as it uses colons...

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The same issue is also discussed in #5471.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I disagree that Rackspace does not have a resolution to their problem. Their own statement in the latest developer guide states:

Pseudo hierarchical folders/directories. To take advantage of this feature, the directory marker Objects must also be created to represent the appropriate directories. The following additional Objects need to be created. A good convention would be to create these as zero or one byte files with a Content-Type of application/directory.

Therefore I expect them to store the files using the marker objects they propose themselves. Eat your own dog food.

@cyberduck
Copy link
Collaborator Author

2d18520 commented

This is Rackspace's response to your last comment:

We've run into this issue with other third-party products as well, and as I mentioned before the issue lies with the fact that everyone has a different standard for the directory objects. If we were to automatically add them, the container would immediately become incomprehensible both to users of our control panel and to all of the other third-party programs that utilize Cloud Files. That's why we suggest that the objects be implemented by the third-party applications themselves; that way the changes will only affect users of that particular program and their implementation will not interfere with others.

I can forward you their admin's email address if you're willing to hash it out with them, otherwise it looks like I'll be writing my own solution similar to the OP for downloading the access logs.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I was in contact with engineering at Rackspace as well. They are now supporting an alternate way to enumerate containers using a delimiter that we can possibly implement which should resolve the issue.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In 5fb15b1.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please test this using the latest snapshot build available.

@cyberduck
Copy link
Collaborator Author

2d18520 commented

Just tried with build 8162, the access log folder is still empty. :-(

Cyberduck log for connecting, accessing root, then accessing .CDN_ACCESS_LOGS:

GET /v1.0 HTTP/1.1
x-auth-user: ************
x-auth-key: ********************************
Host: auth.api.rackspacecloud.com
Connection: Keep-Alive
User-Agent: Cyberduck/4.0b9 (Mac OS X/10.6.5) (i386)
HTTP/1.1 204 No Content
Date: Thu, 30 Dec 2010 15:18:26 GMT
Server: Apache/2.2.3 (Mosso Engineering)
X-Storage-Url: https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_********************************
X-Storage-Token: ********************************
X-CDN-Management-Url: https://cdn.clouddrive.com/v1/MossoCloudFS_********************************
X-Auth-Token: ********************************
X-Server-Management-Url: https://servers.api.rackspacecloud.com/v1.0/************
Content-Length: 0
Connection: close
Content-Type: application/octet-stream

GET /v1/MossoCloudFS_********************************?format=xml HTTP/1.1
X-Auth-Token: ********************************
Host: storage101.dfw1.clouddrive.com
Connection: Keep-Alive
User-Agent: Cyberduck/4.0b9 (Mac OS X/10.6.5) (i386)
HTTP/1.1 200 OK
X-Account-Object-Count: 11878
X-Account-Bytes-Used: 142646861026
X-Account-Container-Count: 3
Content-Length: 396
Content-Type: application/xml; charset=utf8
Date: Thu, 30 Dec 2010 15:18:27 GMT
Connection: keep-alive

GET /v1/MossoCloudFS_********************************/.CDN_ACCESS_LOGS?format=xml&path=&delimiter=%2F HTTP/1.1
X-Auth-Token: ********************************
Host: storage101.dfw1.clouddrive.com
Connection: Keep-Alive
User-Agent: Cyberduck/4.0b9 (Mac OS X/10.6.5) (i386)
HTTP/1.1 200 OK
X-Container-Object-Count: 1267
X-Container-Bytes-Used: 1270906
Content-Length: 86
Content-Type: application/xml; charset=utf8
Date: Thu, 30 Dec 2010 15:18:30 GMT
Connection: keep-alive

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The previous fix was faulty as it still used the path parameter. In 2806cfd.

@cyberduck
Copy link
Collaborator Author

2d18520 commented

Woot. Just confirmed this is fixed in 8197. Thanks for your work on this!!!

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug cloudfiles Rackspace CloudFiles fixed
Projects
None yet
Development

No branches or pull requests

2 participants