Often there is no simple way to get the files from kaggle to a remote server. While previously I had used either a cookies extension or a python command line module that allowed me to specify the competition, neither of these work efficiently or at all for various reasons. I had been meaning to write a script to do this for some time now had never gotten around to it. Finally I did it and here it is posted, this is also incredibly useful for downloading files that are locked away by login/password on some site. For this, I am using RoboBrowser.
The script itself consists of logging in -> getting competition download files -> downloading files.
Probably the easiest part of this was figuring out how to login with RoboBrowser. Simply open a kaggle.com/account/login browser page with RoboBrowser, fill the forms in and then submit.
browser = RoboBrowser(history=True) base = 'https://www.kaggle.com' browser.open('/'.join([base, 'account/login'])) login_form = browser.get_form(action='/account/login') login_form['UserName'] = username login_form['Password'] = password browser.submit_form(login_form)
Getting competition download files
This part as well was easy. Consists of going to competition page, get_links and then only care about ones that end in ‘.zip’ (could add other types or only certain file sizes as well in this part).
browser.open('/'.join([base, 'c', competition, 'data'])) files =  for a_href in browser.get_links(): if '.zip' in a_href.text: files.append(a_href)
Downloading the files
Probably the hardest part was figuring out how to download the files with RoboBrowser since I couldn’t find out any obvious method or way to get them. Figured out the best way to do this (if there’s a better way let me know!) is by using browser.session.get, and stream=True and then basically writing the stream.
for f in files: request = browser.session.get(base + f.attrs['href'], stream=True) with open(f.attrs['name'], "wb") as zip_file: zip_file.write(request.content)
python3 main.py –competition draper-satellite-image-chronology –username my_username –password my_password