What is KindleEar?
KindleEar is a free webapp that running on Google App Engine, it aggregates feeds and deliver the generated docs to your Kindle periodically.
- Custom rss
- Support full text rss and summary rss
- Multi-account management, which means that it supports several kindle devices
- The magazine format MOBI with images
- Automatically daily scheduled delivery
- Powerful email service
- Integration with Evernote/Pocket/Instapaper etc
- All free. Not only this application is free, but GAE provides free hosting space, you do not need to pay a penny.
- Generate fine magazine format books. Sample as follows (the left one is on kindle3, the right two are on kindle touch):
Dows this application require its own server?
This application is hosted on Google App Engine (GAE), GAE has free quotas enough for small applications like KindleEar, you don't need to pay a penny to run KindleEar in GAE, it's fine if you don't own a server(VPS or other infrastructure).
Remember: GAE(Google App Engine) is not GCE (Google Computer Engine), GAE is free forever (with limit of quotas of resources), GCE is free only within 60 days trial. you don't need to start a GCE trial.
How to deploy my own delivery server?
The formal way:
- Download the latest version of KindleEar on github and then extract it to somewhere, for example C:\KindleEar (we'll use it in the following steps).
- You need a google account, please visit the Google Developers Console page. Use your gmail account to login, click on "Create an empty Project", follow steps to create a GAE application.
- Install Python 2.7.x. Download the latest 2.7.x source release and then install it. Do not install a Python 3.x version, because GAE does not support it now.
- Install GAE SDK, which will be used to upload a GAE program. In section on this page, click "Google App Engine SDK for Python", choose your platform, click to download the corresponding version, then install, all the way "next" without words.
- Go to the previously downloaded KindleEar directory (e.g. C:\KindleEar), use a text editor to open app.yaml and module-worker.yaml file, modify the word 'kindleear' in the first line to your GAE application name (e.g. kindleear123), then save and exit.
- Modify variables 'SRC_EMAIL' and 'DOMAIN' and 'TIMEZONE' in config.py in same directory of app.yaml. You can modify other variables if you want and understand what it means.Using the default values are recommended.
- Start upload KindleEar now. You can use Google App Engine Launcher GUI or command line. Because Launcher is not stable, we'll use the command line in Windows.
- Followed by clicking: Start - Run - type cmd, it will open a command line window
- Change current directory to App Engine SDK installation directory, example:
cd C:\Program Files\Google\google_appengine
- Enter 'upload' command:
C:\python27\python.exe appcfg.py update C:\KindleEar\app.yaml C:\KindleEar\module-worker.yaml
- Enter the gmail account and password (password does not display in screen).
- Enter command:
C:\python27\python.exe appcfg.py update C:\KindleEar
- After successfully uploading KindleEar, use your browser to open the site: http://kindleear123.appspot.com (example only, kindleear123 should be changed to the name of your GAE application). If everything goes fine, then go to the last step.
- Add your gmail account to your kindle email whitelist. You should know how to do it already.
- Then what? Enjoy your own delivery service now!
- Tips: the initial username of application is admin, default password is admin too.
The easy way(recommended):
- Download KindleEar and extract it, get a GAE account, create an application.
- Download uploader. (for windows user only)
- Extract 'uploader' to a directory, move the folder of KindleEar into the folder of 'uploader' and rename it to kindleear, then double-click 'uploader.bat' to upload.
- Tips: the initial username of application is admin, default password is admin too.
Why do I not receive the deliveries?
There are many possible reasons:
- You haven't enabled the "Enable deliver custom rss" option, this option is at the bottom of the setting page.
- You haven't added your gmail address to Amazon's 'Approved Personal Document E-mail List'.
- The content of the rss link isn't a valid XML format.
- For those feeds generated by Feedburner, you can append a query string '?format=xml' to end of url, for example: http://xx.com/feed?format=xml.
What is full text rss?
Full text rss means that it contains all the contents of the articles in the rss, while summary rss contains only brief descriptions of the articles. You can check it by opening the link of the rss.
Can full text rss be processed as summary rss? What about the reverse?
Full text rss can be treated as summary rss. In this case, program will ignore the content in the rss xml file and fetch full text from the original link. It costs much time which will result the decreasing of supported rss quantities. If summary rss is processed as full text rss, articles sent to your kindle would still have brief descriptions only.
How to deal with weekly subscription?
Some rss are updated weekly. For this type of rss, you can touch a new python file in the 'books' directory with variable 'deliver_days' setted to a day.
Can KindleEar deliver news to my kindle more than once a day?
The answer is yes, follow the next steps to make it come true:
- Add a python file to the directory of 'books'.
- Set 'deliver_times' using a list in the new python file.
deliver_times=[6,14,22] #6:00,14:00,22:00 deliver three times a day
- Optional: set 'oldest_article' to the number of interval seconds between two deliveries.
How to use the url filter?
Url filter is not for beginners, you need to know regular expression and python. Urls in the list would not be downloaded for saving time.
What can I do if the gfw of China blocked some rss?
Change the link to use https protocol.
How to put the cover or the masthead images in the same directory when I add a customize book?
Set the variable mastheadfile and coverfile to a path, for example:
mastheadfile = 'books/xxxx.gif'
coverfile = 'books/xxxx.jpg'
How to deal with the SSL error problem when I upload KindleEar?
Some friends in China would encounter "urllib2.URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>". To solve this problem, try to add a new line "188.8.131.52 appengine.google.com" to the hosts file.
Why some articles have garbled characters?
For custom rss and the built-in books without encoding specified, program detects encoding automatically by using http header, html meta or the chardet module of python. Howerver, the result is not 100% accurate. If the encoding detecting is wrong, the article may be garbled. Because this situation is very rare, KindleEar only supplies an not very elegant solution: manually change the chardet-detect encoding saved in the database. Steps as follows:
- Login appid.appspot.com by using account 'admin'(assuming your application name is appid).
- Visit 'https://appid.appspot.com/dbviewer'. Check the encoding of the website whose articles are garbled in the UrlEncoding section. If the encoding is wrong, take down its ID.
- Now visit 'https://appid.appspot.com/dbviewer?action=modurlenc&id=dbid&feedenc=fenc&pageenc=penc'
Before you press enter, modify some vaiables:
- dbid: the ID you just took down.
- fenc: new correct FeedEncoding.
- penc: new correct PageEncoding.
- Or you can change them in 'Datastore viewer' in the GAE backend.
- You can get the correct encoding by reading source code of a webpage.
- If you cannot find the website in page 'dbviewer', it means KindleEar has never detected the encoding of any page of this website, http header or html meta is used only. In this case, you can set the variable 'ALWAYS_CHAR_DETECT' in 'config.py' to True.
- If the article still has garbled characters after setting 'ALWAYS_CHAR_DETECT' to True, it means that encoding detected by chardet is wrong and doesn't match with the http header or the html meta. The final attempt is to set 'TRUST_ENCODING_IN_HEADER_OR_META' to True.
- With all the solutions tested and the problem still exists, please add the rss to a python file in the folder 'books' with encoding specified.
What can I do if I forget the password?
If you are a non-administrator user, and you forget the password, you can ask the administrator to change your password. If the administrator password was forgot, you can login appengine.google.com, select "Datastore Admin" on the left, delete all KeUser data, all the account information will be deleted. And then re-use admin/admin to login in your application . Did not find "Datastore Admin"? You can activate it in the "Application Settings".
If you know python, I'm sure that you can find other solutions.
How to use mail service email@example.com?
You'll have uncountable mail addresses like firstname.lastname@example.org as long as xxx is valid ascii string after you have deployed KindleEar(appid is your application id).
- Before you can enjoy the feature, please add email addresses to the whitelist in menu 'Advance'|'White List'. A asterisk '*' means allow anyone to trigger it, or only allow some people by adding their emaill addresses like 'email@example.comfirstname.lastname@example.org'.
- KindleEar will convert the text of the email received from email@example.com as an attachment and then send it to your registered email address. If there are only hyperinks in the email, KindleEar will first fetch the url, generate a book and then send it.
- Add flag !links to end of subject to indicate KindleEar crawl all links in email and transfer to your Kindle.
- Add flga !article to end of subject to indicate KindleEar ignore all links in email.
- The generated book will be sent to admin's mailbox by default. Use firstname.lastname@example.org(Note: there're two underscores) if you want the book to be sent to other registered users.
- Send emails with the download link of an ebook to email@example.com or firstname.lastname@example.org will trigger a task to download the ebook and then diliver it to kindle directly.
Refer to Mail Python API Overview to get to know the list of allowed postfix of ebooks.
- Send emails to email@example.com or firstname.lastname@example.org will trigger a delivery by hand.It equals to the 'deliver now' button in 'Setting' Menu if the subject is empty or is 'all'.Deliver specific books by adding the name of the book, seperate with commma if there is more than one.
Why I can't lookup words in the delivery books after I chosen 'Merge books into one'?
With the option 'Merge books in one' checked, KindleEar will set the metadata of the book as Custom Rss. To fix this problem, you have to set the language attribute in the 'Custom Rss Setting' section manually.
How to deal with those websites that need login before I can view the articles?
If you want to crawl a website that needs login, you have to add a python file to the folder 'books'. Refer to sample books if you don't know how to write it.
needs_subscription : Set to True
login_url : The URL for login page
form_4_login : Optional, if not set, KindleEar will guess which form in the page is used to login. Set to an integer indicate the sequence number (where 0 is the first) of the form in the page. Set to a string indicate the name of the form, you can use CSS selector: '#' for id and '.' for class, string only will match any of it(name/id/class).
- Then login to KindleEar. In the 'Feeds' page, a new 'Login info' button will appear on the right side of the book which needs_subscription setted to True in the 'Subscripted' section. Click on it and then fill in your account name and password.
Note2: for some special site, you may have to overwirte the 'login' function which inherits from the 'BaseFeedBook' class.
Note3: the password will be stored encrypted on GAE, the key for encryption is a random string for each account of KindleEar, but the key is also stored on GAE, so it's not absolutely safe, please keep this in mind.
Can I delete/hide books in Chinese?
If you are not Chinese, you can delete some *.py files in directory 'books'.
'__init__.py' and 'base.py' are required, please don't remove them.
If you deploy the application via uploader, uploader can help you to delete books which language is Chinese.
What's the meaning of the "Separate" option for the built-in books?
If you choose this option, the book you chosen will be sent to kindle separately even if the option "Merge books into one" is chosen.
How to solve it if website shows "internal server error" exception.
Open appengine.google.com, choose your application, open "Logs", if have “NeedIndexError: no matching index found.” exception, then you can execute command in your computer.
python appcfg.py update_indexes YourKindleEarFolder/
if have “UnknownQueueError” exception, execute:
python appcfg.py update_queues YourKindleEarFolder/
After command be executed, please wait some miniutes for GAE generate index for you.
If the problem still exist, you should deploy KindleEar again.
How to use the feature 'Save to Pocket'?
The Pocket is a popular 'Read It Later' service.
The service allows the user to save an article or web page to the cloud for later reading. The article is then sent to the user's Pocket list (synced to all of their devices) for offline reading. Pocket removes clutter from articles and allows the user to adjust text settings for easier reading.
Now you can save an article of your Kindle to your pocket's account, and then you can read it in other devices later.
Follows steps to enjoy this feature:
- Get an account of pocket;
- Click [Authorize] of section 'Save to Pocket' in KindleEar's page 'Advance' |' Share';
- Choose "Append hyperlink 'Save to Pocket' to article" in KindleEar's page 'Advance' | 'Share'.
- Authorization use OAuth v2 technology, only Pocket knows your username and password;
- This application requests 'Add' permission only, asks for neither 'Modify' nor 'Delete', can't retrieve your article list too.
- The Pocket's access token stored in your database, KindleEar application of other guys do not have permission to access your Pocket account;
- If you worry about privacy and security, you can apply an new developer Consumer Key in Pocket website, and then change variable POCKET_CONSUMER_KEY in config.py.
- If you want KindleEar deliver your unread articles to KinleEar, you can find a RSS address in Pocket page, and add it to list of subscription of KindleEar.
- Both username and password are requested to active the feature 'Save to Instapaper', but the password is encrypted before be saved to database.
My logs shows error 'wrong SRC_EMAIL'?
Go to your GAE console, in 'Settings' | 'Application settings' page, add your email into field 'Email API authorized senders'.